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FOREWORD 


The  Training  and  Simulation  Technical  Area  of  the  Army  Research  Insti¬ 
tute  for  the  Behavioral  and  Social  Sciences  (ARI)  has  supported  the  Army 
Project  Manager  for  Training  Devices  (PM  TRADE)  in  its  research  program  to 
develop  new  methods  for  designing  and  procuring  maintenance  training  devices 
The  Army  Maintenance  Training  and  Evaluation  Simulation  System  (AMTESS) 
project  is  an  effort  to  build  modularized  maintenance  training  devices  and 
also  to  develop  new  approaches  to  the  maintenance  trainer  procurement 
process . 

This  report  is  a  review  of  the  developmental  history  of  the  AMTESS 
project.  It  documents  the  activities  of  the  agencies  involved  in  the  pro¬ 
ject  and  traces  how  these  activities  led  to  specific  training  device  hard¬ 
ware  designs.  It  also  documents  how  the  evaluations  of  the  prototype  de¬ 
vices  proceeded.  This  documentation  of  "lessons  learned"  from  AMTESS  can 
be  used  to  improve  future  procurements  of  maintenance  training  devices. 

This  report  may  be  used  by  the  training  community  to  help  avoid  pre¬ 
vious  mistakes  as  well  as  to  recognize  procedures  that  work.  It  can  sensi¬ 
tize  the  representatives  of  interacting  agencies  in  the  training  device  pro¬ 
curement  process  to  pitfalls,  which,  if  avoided,  will  facilitate  the  process 
In  addition,  this  report  can  be  used  by  researchers  to  help  them  prepare  for 
the  contingencies  they  will  encounter  in  future  training  device  evaluations. 


EDGAR  M.  JOHNSON 
Technical  Director 


ARMY  MAINTENANCE  TRAINING  AND  EVALUATION  SIMULATION  SYSTEM  (AMTESS) 
DEVICE  DEVELOPMENT  AND  FEATURES 


EXECUTIVE  SUMMARY 


Requirement: 

This  report  includes  a  discussion  of  (1)  processes  in  the  development  of 
two  Army  Maintenance  Training  and  Evaluation  Simulation  System  (AMTESS) 
breadboard  (first-run)  generic  training  devices,  and  (2)  the  specific  fea¬ 
tures  on  each  device.  One  goal  of  this  report  is  to  document  several  years 
of  interaction  among  government  agencies,  device  contractors,  and  evaluation 
contractors  for  the  purpose  of  elucidating  strengths  and  weaknesses  in  the 
process  of  developing,  acquiring,  and  testing  generic  training  simulators. 

A  second  goal  is  to  identify  strengths  and  weaknesses  in  specific,  generic 
training  device  features.  A  third  goal  is  to  offer  recommendations,  based 
on  AMTESS  experiences,  for  similar  future  efforts.  The  AMTESS  project  may  be 
seen  as  a  case  study  in  training  simulator  development. 


Procedure: 

First,  a  number  of  documents  were  examined  to  develop  a  chronology  (1977 
to  1983)  of  government  and  contractor  participation  in  the  AMTESS  project. 
Second,  key  participants  representing  the  government  and  contract  firms  were 
personally  interviewed  for  their  opinions  about  the  AMTESS  development  and 
testing  process  and  specific  device  features.  Data  from  the  first  and  second 
sources  concerning  AMTESS  project  development  were  collated,  displayed  in 
figures,  and  discussed.  Next,  opinions  about  device  features  were  summarized. 
Finally,  experts  rated  specific  device  features,  and  ratings  were  subjected 
to  a  multi -attribute  utilities  analysis. 


Findings : 

An  examination  of  AMTESS  documents  and  interviewee  opinions  revealed 
that  problems  arose  during  AMTESS  device  development,  acquisition,  and  test¬ 
ing.  Most  of  these  problems  centered  on  the  need  for  (1)  more  frequent, 
more  precise  communications,  (2)  clearer  definitions  of  the  explicit  respon¬ 
sibilities  of  each  agency,  both  government  and  contractor,  (3)  more  explicit 
mechanisms  for  quality  control  of  the  devices,  (4)  greater  anticipation  of 
disruption  contingencies,  and  (5)  the  need  for  more  high-level  administrative 
and  financial  resources  appropriate  to  the  responsibilities  imposed  on  pro¬ 
gram  personnel  than  were  available  during  the  AMTESS  project. 

Two  devices  were  tested  during  AMTESS:  one  developed  by  the  Seville 
Research  Corporation  and  the  Burtek  Corporation  and  the  other  by  the  Grumman 
Aerospace  Corporation.  Interviewees  and  experts  felt  that  many  device  fea¬ 
tures  were  based  on  sound  instructional  concepts.  Implementation  problems, 
however,  especially  disrepair,  plagued  both  devices.  Some  of  the  most 


valued  features  of  the  Seville/Burtek  device  were  its  high  fidelity  3D 
module  and  comprehensive  student  performance  record.  The  3D  module  with 
its  high  fidelity  parts  and  easy  access  design  was  especially  valued  on 
the  Grumman  device. 

Recommendations  are  offered  concerning  future  device  development  and 
evaluation  efforts.  These  recommendations  center  around  more  frequent  com¬ 
munication  among  participants  and  greater  specificity  concerning  partici¬ 
pants'  responsibilities. 


Use  of  Findings: 

The  findings  of  this  report  will  be  of  interest  to  the  training  device 
development  field.  This  report  will  be  useful  in  planning  the  development 
of  maintenance  training  simulators,  particularly  generic  maintenance 
simulators. 
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INTRODUCTION 


The  Army  Maintenance  Training  and  Evaluation  Simulation  System  (AMTESS) 
represents  an  attempt  to  define  new  concepts  in  the  area  of  maintenance 
training.  Traditionally,  Army  students  receive  maintenance  training  on 
operational  equipment.  However,  this  type  of  training  may  be  counterpro¬ 
ductive  for  several  reasons.  Often,  training  equipment  is  unlike  field 
equipment,  and  training  is  unlikely  to  generalize  to  field  equipment.  Fre¬ 
quently,  faults  used  for  diagnostic  training  are  only  a  small  subset  of 
malfunctions  found  in  the  field.  In  addition,  equipment  may  be  damaged 
during  training,  or  students  may  be  injured  during  operation  of  hazardous 
equipment.  Finally,  availability  of  operational  equipment  is  limited. 

The  AMTESS  project  has  sought  to  develop  maintenance  training  devices  to 
alleviate  some  of  the  problems  encountered  in  actual  equipment  training. 

As  stated  in  the  Request  for  Proposal  (RFP)  for  the  first  phase  of  the 
AMTESS  project,  device  design,  the  AMTESS  project  had  three  objectives: 

1.  Conceptually  design  a  modular  maintenance  trainer  that  could  be 
used  for  hands-on  training  and  testing  at  the  institution  and 
unit  levels. 

2.  Provide  economic  impact  assessment  of  the  effect  of  the  intro¬ 
duction  of  AMTESS  devices  into  the  Army  maintenance  program. 

3.  Fabricate  a  breadboard  (first-run)  working  model  for  evaluation. 

The  initial  RFP  also  included  several  features  which  should  characterize 
AMTESS  devices: 

•  Adjunct  to  primary  instruction 

t  Individualized,  self-paced  instruction 

•  Hands-on,  actual  practice  capability 


•  Easily  producible 

•  Modular,  with  easy  interchange  of  modules 

•  Flexible,  adaptable  to  changes  in  school  curricula, 
including  an  editing  system  for  school  personnel  to 
use  for  updating 

•  Instructor  and  student  stations 

•  Easily  transportable 

•  Complete  software 

Automotive  and  missile  maintenance  tasks  were  to  be  taught  by  the  first 
AMTESS  devices,  but  more  devices  were  envisioned  in  the  "family  of  maintenance 
trainers."  Those  future  devices  included  aviation,  electronics,  heavy  equip¬ 
ment,  armor  and  artillery  maintenance  training  simulators. 

The  AMTESS  project  was  sponsored  by  the  Project  Manager,  Training 
Devices  (PM  TRADE).  PM  TRADE'S  AMTESS  developmental  program  consisted  of 
two  phases.  During  the  first  phase,  new  training  device  concepts  were 
written.  During  the  second  phase,  breadboard  devices  were  constructed. 

Phase  I  of  the  AMTESS  development  program  was  aimed  at  the  definition 
of  conceptual  designs  for  a  modular  maintenance  trainer.  To  arrive  at  a 
conceptual  design,  five  major  tasks  were  performed  by  each  of  four  device 
contractors: 

1 .  Task  analysis, 

2.  Training  requirements  analysis, 

3.  Fidelity  requirements  analysis, 

4.  Design  of  a  modular  system,  and 

5.  Assessment  of  potential  economic  impact. 

Phase  II  of  the  AMTESS  research  and  development  program  involved  the 
fabrication  of  breadboard  models  of  two  of  the  four  designs  developed  during 
Phase  I.  The  Grumman  Aerospace  Corporation,  and  a  consortium  of  Seville 
Research  Corporation  and  Burtek,  Inc.  produced  these  breadboard  models. 


The  Army  Research  Institute  ( AR I )  sponsored  the  project  which  included 
AMTESS  device  evaluation.  This  project  was  called  SIMTRAIN  I.  SIMTRAIN  I 
included  three  tasks.  The  first  task  involved  the  identification  of  methods 
for  evaluating  device  requirements  (Heeringa,  Baum,  holman,  &  Peio,  1982; 
Kane  &  Holman,  1982)  and  development  of  new  methods  of  evaluating  effective¬ 
ness  (Eberts,  Smith,  Dray,  &  Vestewig,  1982;  Klein,  1982;  Tufano  &  Evans, 
1982). 

Task  2  involved  the  specification  of  simulation  fidelity  (Baum,  Riedel, 
Hays,  4  Mirabel  la,  1982;  Hays,  1980;  Hays,  1981).  Task  3  involved  a  field, 
transfer-of-training  evaluation  of  the  two  AMTESS  breadboard  simulators 
(Unger,  Moyer,  Cole,  &  Swezey,  1983). 

The  present  report  is  an  evaluative  effort  in  the  SIMTRAIN  II  project. 
(SIMTRAIN  II  continues  the  AMTESS  evaluation  and  includes  systematic  exten¬ 
sions  of  previous  AMTESS  experiments.)  One  goal  of  this  report  is  to  docu¬ 
ment  several  years  of  interaction  among  government  agencies,  device 
contractors,  and  evaluation  contractors  for  the  purpose  of  elucidating 
strengths  and  weaknesses  in  the  process  of  generic  training  simulator 
development  and  testing.  A  second  goal  is  to  identify  strengths  and  weak¬ 
nesses  of  specific  AMTESS  device  features.  A  third  goal  is  to  offer  guide¬ 
lines  for  future  device  development  efforts. 

This  report,  has  three  sections.  Section  I  describes  the  development 
of  the  AMTESS  project.  The  procedures  used  in  gathering  a  relevant  data 
base  are  discussed.  Important  events  and  relationships  in  the  history  of 
AMTESS  are  displayed  graphically  and  discussed. 

Section  II  describes  specific  features  of  two  breadboard  devices. 

The  data  base  for  this  section  includes  the  personal  opinions  of  personnel 
representing  nine  organizations  involved  in  AMTESS  and  quantitative  ratings 
by  a  small  panel  of  device  experts. 

Section  III  summarizes  all  the  findings  and  offers  suggestions  for 
future  efforts  in  the  development  of  generic  maintenance  simulators. 


I.  RECORDING  THE  HISTORY  OF  THE  ARMY  MAINTENANCE  TRAINING 
AND  EVALUATION  SIMULATION  SYSTEM  (AMTESS) 


A.  Procedures 

Information  about  the  history  of  AMTESS  came  from  two  sources:  documents 
and  structured  interviews. 

1.  Documents.  A  large  number  of  formal  and  informal  documents  was 
examined.  Appendix  A  contains  a  list  of  all  documents  reviewed.  Informa¬ 
tion  extracted  included  the  dates  and  participants  in  AMTESS  events. 

2.  Structured  interviews.  Key  personnel  representing  the  government, 
device  contractors,  and  the  evaluation  contractor  (SAI)  were  interviewed. 

Two  interviewers  conducted  each  interview. 

Table  1  presents  a  list  of  all  interviews  conducted.  It  gives  descriptive 
information  about  each  interview,  identifies  each  interviewee,  and  presents 
the  date  and  location  of  each  interview. 

Eighteen  people  representing  nine  organizations  were  interviewed.  Those 
nine  organizations  were  judged  to  be  the  most  heavily  Involved  in  the  AMTESS 
program.  Personnel  representing  other  organizations  (Honeywell,  Hughes, 
USAMMC&S ,  and  USALOGCEN,  for  example)  were  not  interviewed  because  their 
scope  of  involvement  in  AMTESS  was  less  than  that  of  those  organizations 
interviewed,  and  time  and  financial  constraints  dictated  that  not  every 
participant  could  be  interviewed. 

The  structured  interview  protocol  was  written  by  SAI.  This  protocol 
contained  three  sections  of  questions.  The  sections  concerned  the  history 
of  AMTESS,  front  end  analysis,  and  device  features.  Appendix  B  contains  a 
copy  of  the  protocol . 

Each  interview  began  with  introductions  of  interviewee  and  the  two 
interviewers.  The  interviewee  was  told  that  the  purpose  of  the  interview 


TABLE  1.  INTERVIEW  INFORMATION 


ORGANIZATION 

NUMBER  OF 
INTERVIEWEES 

INTERVIEW 

LOCATION 

PM  TRADE 

Orlando,  FL 

1 

Ft.  Bliss 

ARI,  Simulation 

Systems  Technical  Area 
Alexandria,  VA 

2 

ARI 

Ft.  Bliss 

USAOC&S 

Aberdeen,  MD 

2 

Aberdeen 

USAADS 

Ft.  Bliss,  TX 

1 

Ft.  Bliss 

TRADOC,  ATSC 

Ft.  Eustis,  VA 

1 

Ft.  Eustis 

Grumman 

Bethpage,  NY 

2 

Bethpage 

Seville  Research  Corp. 
Pensacola,  FL 

2 

Pensacola 

Burtek,  Inc. 

Tulsa,  OK 

3 

Tulsa 

Science  Applications,  Inc. 

4 

McLean 

McLean,  VA 


INTERVIEW 

DATE 


24  May  1983 


9  June  1983 
23  May  1983 


6  May  1983 


23  May  1983 


3  May  1983 


14  June  1983 


5  May  1983 


16  May  1983 

17  May  1983 
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was  to  learn  about  the  AMTESS  development  process  so  similar  future  efforts 
might  be  improved.  The  primary  interviewer  then  conducted  the  interview  and 
took  notes  as  much  as  possible.  The  second  interviewer  took  extensive  notes 
and  occasionally  requested  clarification.  Interviewees  were  asked  to  respond 
in  depth  to  those  questions  about  which  they  were  knowledgeable  and  to  give 
no  comment  if  unfamiliar  with  the  topic.  Care  was  taken  to  put  interviewees 
at  their  ease.  Average  interview  duration  was  about  two  hours. 

Individuals  interviewed  were  assured  that  their  comments  would  not  be 
attributed  to  them  by  name.  Interviewees'  opinions  are  not  to  be  construed 
as  the  official  position  of  their  employers.  Interviewees'  opinions  reflect 
personal  judgments  based  on  experience  in  the  AMTESS  program. 

After  each  interview,  the  interviewers  discussed  the  interviewee's 
response  to  each  question.  During  this  time,  the  interviewers  reached 
agreement  about  the  interviewee's  responses  and  drew  connections  between 
responses. 

3.  Data  Reduction  and  Display  Procedures.  From  the  extensive 
chronology  extracted  from  AMTESS  documents,  key  events  were  selected. 

This  information  was  organized  around  two  figures  (presented  and  discussed 
in  Section  I.B.  of  this  report). 

Information  from  structured  interviews  aggregated  around  five  topics: 
objectives  of  AMTESS,  performance  of  organizations  involved,  coordination 
of  effort,  the  device  evaluation,  and  the  future  of  AMTESS.  All  responses 
were  collated  question-by-question  within  each  issue  and  then  summarized. 

The  discussion  of  structured  interview  responses  is  presented  in  Section  I.C. 
of  this  report. 

Although  interviewees  were  questioned  about  their  front  end  analysis 
activities,  very  little  information  was  obtained.  Most  interviewees  were 
not  involved  in  the  AMTESS  effort  during  the  front  end  analysis  portion  of 
the  project.  Front  end  analysis  is  not  treated  as  a  separate  topic  in  this 
report,  but  will  be  thoroughly  discussed  in  a  subsequent  report. 
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B.  Results:  A  Brief  History  of  the  AMTESS  Effort 


1.  Overview.  This  history  of  AMTESS  begins  in  May  1977  with  the 
proposal  review  for  a  maintenance  training  study  and  ends  in  June  1983 
with  the  conclusion  of  the  first  AMTESS  device  evaluation.  During  these 
six  years,  five  major  contracts  were  awarded.  Five  government  agencies 
and  six  contract  firms  were  key  players.  These  organizations  were  based 
in  New  York,  Maryland,  Virginia,  Florida,  Arkansas,  Texas,  Oklahoma,  and 
Minnesota. 

Figure  1  presents  the  major  events  in  the  AMTESS  project  and  includes 
an  overview  of  the  relationships  among  the  principal  participants. 

The  Study  Advisory  Group  was  organized  by  PM  TRADE  and  originally 
included  representatives  from  PM  TRADE  and  USATSC.  Later,  representatives 
from  USALOGCEN,  USAADS,  and  USAOC&S  were  added.  As  shown  in  Figure  1,  the 
Study  Advisory  Group  awarded  to  the  Seville  Research  Corporation  a  contract 
to  investigate  the  concept  of  using  simulators  in  maintenance  training.  From 
1977  to  1979,  PM  TRADE  developed  the  AMTESS  concept  with  the  help  of  the 
Study  Advisory  Group  and  with  input  f rom . Sevi 1 1 e . 

Late  in  1979,  PM  TRADE  awarded  Phase  I  AMTESS  contracts  to  four  firms: 
Grumman  Aerospace  Corporation,  Honeywell  Systems  and  Research  Center,  the 
Seville  Research  Corporation,  and  Hughes  Aircraft  Company.  Figure  1  shows 
this  branching  out  to  the  four  contractors.  Each  firm  was  to  conduct  a 
front  end  analysis  and  then  posit  a  conceptual  design  for  a  generic  AMTESS 
device.  The  AMTESS  device  was  generic  in  the  sense  that  it  was  to  include 
one  core  component  and  various  3D  modules  relating  to  different  MOSs. 

The  Phase  I  front  end  analysis  included  three  main  components:  task 
commonality  analysis  for  selected  automotive  and  missile  MOSs,  training 
requirements  analysis,  and  fidelity  analysis.  Other  requirements  in  Phase  I 
included  life  cycle  cost  estimates  and  a  preliminary  system  design.  All 
four  contractors  submitted  Phase  I  final  reports  by  July  1980  as  shown  in 
Figure  1. 
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As  shown  in  Figure  1,  several  problems  occurred  during  Phase  I. 

These  problems  remained  unsolved  and  carried  over  into  Phase  II.  The 
problems  included  lack  of  a  cooperative  agreement  among  participants, 
and  poorly  documented  school-contractor  contacts. 

Phase  II  of  AMTESS  began  in  September  1980  with  PM  TRADE'S  contract 
awards  to  Grumman  Aerospace  Corporation  and  a  team  consisting  of  Seville 
Research  Corporation  and  the  Burtek  Corporation.  In  Phase  II,  these  two 
contractors  were  to  construct  breadboard  models  of  the  devices  they 
designed  in  Phase  I.  These  two  contractors  were  selected,  in  part, 
because  their  device  designs  were  quite  different  from  each  other.  Each 
contractor  would  construct  two  devices,  one  for  each  of  the  two  test 
schools  (USAADS  and  USAOC&S).  The  devices  would  address  the  MOSs  selected 
by  each  school . 

The  selection  of  skills  to  be  taught  by  the  devices  turned  out  to  be 
one  of  the  largest  problems  in  Phase  II.  As  shown  in  Figure  1,  a  problem 
in  this  regard  occurred  after  Phase  II  proposals  had  been  submitted,  but 
before  Phase  II  award.  Three  of  the  five  MOSs  from  which  skills  had  been 
drawn  were  dropped  from  the  curriculum  at  USAOC&S.  Thus,  at  the  time  of 
award,  the  government  could  not  specify  a  skill  list  for  the  contractors. 
Several  meetings  occurred,  and  by  December  1980,  the  tasks  were  specified. 
However,  Figure  1  shows  that  a  major  problem  with  the  task  selections  still 
existed:  the  tasks  selected  were  different  for  each  device,  thereby  making 
invalid  any  experimental  comparisons  that  might  have  been  made  between  them. 
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Shortly  after  the  Phase  II  award,  as  shown  in  Figure  1,  ARI  awarded  to 
Honeywell  a  contract  called  S1MTRAIN  I.  Honeywell's  involvement  in  SIMTRAIN  I 
is  largely  out  of  the  scope  of  this  report,  but  Honeywell's  subcontractor,  SAI, 
had  the  task  of  conducting  an  empirical  evaluation  of  the  Grumman  and  Seville/ 
Burtek  devices.  SAI,  with  ARI's  guidance,  began  developing  during  the  time 
of  device  development  (in  mid-1981)  test  plans  and  actual  performance  tests 
for  use  in  the  evaluation.  These  simultaneous  and  related  efforts  are  shown 
in  Figure  1. 
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Finally,  from  mid  to  late  1982,  the  devices  arrived  at  the  test  schools. 
Figure  1  again  shows  a  problem,  this  time  with  device  acceptance.  The  tasks 
taught  by  two  devices  did  not  match  either  the  tasks  selected  in  December 
1980,  the  tasks  used  by  the  test  schools  and  SAI  to  develop  performance  tests, 
or  the  school's  conventional  curriculum  which  was  to  serve  as  control  training. 
At  the  last  minute,  new  performance  tests  were  developed  and  the  evaluations 
began.  SAI  completed  its  evaluation  of  both  devices  in  June  1983. 

The  following  sections,  I.B.2.  and  I.B.3.,  discuss  the  AMTESS  device 
developmental  process  in  detail. 

2.  AMTESS  Phase  I.  Figure  2  presents  the  11  main  groups  involved  in 
the  AMTESS  project  and  events  critical  to  their  participation.  Figure  2 
shows  the  tasks  performed  by  single  groups  and  illustrates  how  activities 
■of  different  groups  overlapped  in  time.  (Some  groups  such  as  USALOGCEN 
and  USAMMCS  have  not  been  listed  separately  in  Figure  2.  These  agencies 
participated  in  the  various  advisory  groups,  listed  in  Figure  2  as  Advisory/ 
Working  Groups.) 

The  AMTESS  concept  was  developed  during  1977  and  1978.  During  this 
time,  PM  TRADE  sponsored  a  study  to  survey  maintenance  training  devices 
used  by  the  Army,  other  services,  and  private  schools  and  industry.  PM  TRADE 
organized  a  Study  Advisory  Group  (SAG)  in  September  1977  to  oversee  the  study. 
The  Seville  Research  Corporation  conducted  the  survey.  The  main  goal  of  the 
study  was  to  recommend  state-of-the-art  alternatives  to  actual  equipment 
training.  Seville's  final  report  was  submitted  in  September  1978.  Clearly, 
Seville  is  the  one  contract  firm  which  has  been  involved  in  AMTESS  since 
1977,  and  one  of  its  employees  who  was  involved  in  the  survey  study  is  still 
involved  in  AMTESS. 

In  June  1978,  a  group  of  28  government  personnel  met  at  ATSC  to  discuss 
various  points  of  view  about  the  development  of  maintenance  trainers.  Two 
Army  schools,  USAADS  and  USAMMCS,  submitted  a  statement  of  their  needs  and 
requirements  for  a  generic  maintenance  training  device.  According  to  notes 
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from  one  USAADS  employee,  the  schools  wanted  improved  training  devices. 

The  extensive  research  and  development  envisioned  by  several  agencies 
was  not  envisioned  by  the  schools  at  that  time.  Thus,  a  problem  with 
mismatched  goals  might  have  been  born  during  this  time.  The  group 
affirmed  that  PM  TRADE  would  develop  a  request  for  proposal  concerning 
development  of  maintenance  trainers,  and  manage  the  subsequent  contract. 

Accordingly,  in  July  1978,  PM  TRADE  formed  a  coordinating  organization, 
the  Joint  Study  Advisory  Group  (JSAG).  Specific  organizations  invited  to 
participate  were  PM  TRADE,  ARI,  USATSC  and  nine  schools,  USAIS,  USAOC&S, 

USAAMMC&S ,  USASIGS,  USAFAS,  USAARMS,  USAADS,  USATSCH  and  USALOGCEN.  The 
primary  objectives  of  this  group  were  to  develop  the  AMTESS  concept,  develop 
long-range  maintenance  training  plans,  and  develop  the  statement  of  work  for 
AMTESS  Phase  I.  Government  personnel  directly  involved  in  AMTESS  have  changed 
substantially  since  the  first  JSAG  meeting  in  August  1978.  Of  the  16  individuals 
at  that  meeting,  only  two  were  active  through  AMTESS  Phase  II  and  device 
evaluation. 

Later  in  1978,  the  JSAG  was  restructured.  The  new  group  was  called  the 
Joint  Working  Group  (JWG). 

The  Phase  I  contracts  were  awarded  in  September  1979  (about  nine  months 
later  than  planned).  Of  the  11  individuals  on  the  proposal  review  committee, 
again,  only  two  remain  active  in  AMTESS.  Phase  I  contracts  were  awarded  to 
four  contractors:  Grumman,  Hughes,  Honeywell,  and  Seville. 

In  December  1979,  PM  TRADE  authored  a  memorandum  of  understanding 
between  itself  (a  DARCOM  organization)  and  USATSC  (A  TRADOC  organization), 
supported  by  USAOC&S,  USAMMC&S,  USADDS,  and  ARI.  The  responsibilities  of 
each  organization  were  spelled  out  in  general.  It  is  uncertain,  however, 
if  the  document  was  finalized  and  signed  by  all  parties. 

As  mentioned  earlier,  the  goals  of  Phase  I  were:  (1)  to  conduct  a 
front  end  analysis  including  task  commonality  analysis,  training  require¬ 
ments  analysis,  and  fidelity  analysis;  (2)  estimate  life  cycle  system  costs; 


and  (3)  design  a  high  technology  generic  maintenance  trainer.  Personnel  at 
USAADS,  USAOC&S,  USAAS,  and  USAMMC&S  were  asked  to  provide  the  contractors 
with  as  much  information  as  possible  about  their  curricula.  Contractor  and 
school  contacts  during  Phase  I  are  not  well  documented,  and  few  personnel 
remain  to  report  the  details  of  Phase  I  activity. 

It  is  not  the  purpose  of  this  report  to  document  and  discuss  activities 
in  the  front  end  analysis.  A  report  on  AMTESS  front  end  analysis  activities 
is  the  subject  of  another  report  (under  the  SIMTRAIN  II  contract,  Task  2) 
due  in  1984. 

Final  Phase  I  briefings  were  presented  by  all  four  contractors  at 
PM  TRADE  on  March  26,  1980.  Grumman  proposed  a  generic  training  device 
consisting  of  a  2D  module  with  computer,  video  disc,  and  CRTs,  which  linked 
to  a  3D  computerized,  simulated  piece  of  equipment  that  was  scaled  down  in 
size.  Honeywell's  and  Hughes’s  devices  were  also  designed  with  a  video  disc. 
Seville  proposed  a  device  of  2D  component  with  computer,  random  access  slide 
projector,  and  CRT.  Seville's  3D  component  was  to  be  a  full  size  computer- 
activated  piece  of  equipment. 

Final  reports  were  submitted  by  Honeywell,  Seville,  Hughes,  and  Grumman 
in  May,  May,  June,  and  July  1980,  respectively. 

3.  Phase  II  and  Device  Evaluation.  PM  TRADE  developed  the  statement 
of  work  for  Phase  II  in  1979  and  1980.  The  major  tasks  in  Phase  II  were  to 
be  the  construction  and  acceptance  of  breadboard  generic  maintenance  training 
devices.  These  devices  had  been  designed  in  Phase  I. 

Coordination  of  Phase  II  activities  was  the  responsibility  of  PM  TRADE 
and  the  Joint  Working  Group  (JWG).  This  group  was  organized  by  PM  TRADE  in 
late  1978.  Agencies  in  the  JWG  were  PM  TRADE,  ARI,  ATSC,  USALOGCEN,  USAOC&S , 
USAADS,  and  USAAMCS.  Contractor  representatives  were  invited  to  attend  JWG 
meetings  as  appropriate.  The  JWG  coordinated  activities  through  Phase  II, 
the  device  evaluation,  and  remains  active  still. 


In  January  1980,  as  shown  in  Figure  2,  ARI  submitted  a  device  evaluation 
plan  to  PM  TRADE.  The  evaluation  plan  was  complex,  included  80  cells,  and 
was  substantially  simplified  before  implementation. 

The  MOSs  used  in  the  Phase  I  task  commonality  analysis  were  63B,  C,  and 
H  in  automotive  systems,  and  24C  and  K  in  missile  systems.  Plans  made  for 
the  devices  were  based  on  those  MOSs.  Unfortunately,  problems  arose  in 
July  1980,  after  the  Phase  II  proposals  were  evaluated,  but  before  the 
contracts  were  awarded.  The  63B,  C,  and  H  MOSs  were  dropped  from  the  curric¬ 
ulum  at  USAOC&S;  therefore,  the  tasks  to  be  taught  by  the  devices  changed. 

This  meant  that  some  portion  of  Phase  I  device  planning  was  no  longer 
applicable.  Further,  changes  in  school  curricula  meant  that  control  (con¬ 
ventional)  training  had  changed.  Clearly,  a  reassessment  with  respect  to 
device  development  and  evaluation  was  in  order.  Perhaps  at  that  point, 
because  of  the  extensive  changes  that  had  occurred  in  task  selection,  the 
contract  process  should  have  been  delayed  until  the  schools  and  ARI  decided 
on  a  task  list  that  would  fit  with  the  school's  curriculum  and  lend  itself 
to  performance  testing  for  purposes  of  device  evaluation.  Then,  perhaps 
another  task  commonality  analysis  should  have  been  conducted. 

The  AMTESS  Phase  II  contracts  were  awarded  in  September  1980,  before 
the  task  selection  problem  was  solved.  Shortly  after  contract  award,  the 
Phase  II  contractors,  Seville/Burtek  and  Grumman,  met  with  the  government, 
but  did  not  agree  on  task  selection.  Then,  at  a  JWG  meeting  in  November 
1980,  contractors  were  told  that  they  needed  to  add  training  in  the  use  of 
the  STE/ICE  (Standard  Test  Equipment  for  Internal  Combustion  Engines)  device. 
The  task  selection  problem  was  thereby  magnified. 

Contractor  and  school  personnel  finalized  task  selection  by  December  1980. 
For  USAOC&S,  the  Grumman  simulator  was  to  be  based  on  tasks  from  the  63D30  M0S, 
and  Seville/Burtek's  simulator  on  63W10.  For  USAADS,  both  devices  would  be 
based  on  the  24C10  M0S,  but  the  overlap  in  tasks  taught  by  the  device  was 
only  partial.  (Date  of  USAADS  task  selection  is  unknown.)  The  contractors 
and  schools  would  work  closely  together  for  the  next  15  months  of  device 
development. 


AR I  was  informed  of  task  selection  for  each  device  in  December  1980. 
ARI  expressed  reservations  about  the  selection  because  ARI's  device  evalua¬ 
tion  plans,  submitted  to  PM  TRADE  nearly  one  year  earlier,  called  for  a 
comparison  between  the  two  devices  on  measures  of  transfer  of  training 
from  the  simulators  to  the  actual  equipment.  Such  a  comparison  would  be 
invalid  if  the  devices  did  not  teach  the  same  tasks.  ARI  communicated 
their  concern  to  PM  TRADE,  but  PM  TRADE  did  not  insist  that  the  devices 
teach  the  same  tasks. 


Shortly  after  the  task  list  was  specified,  in  March  1981,  ARI  awarded 
to  Honeywell,  with  SAI  as  a  subcontractor,  a  contract  to  support  the  AMTESS 
device  evaluation.  AMTESS  device  evaluation  was  one  task  of  a  large  device 
evaluation  project  called  SIMTRAIN  I.  Honeywell  submitted  a  draft  test  plan 
to  ARI  in  April  1981  taking  into  account  the  changes  in  task  selection  and  • 
removing  experimental  comparisons  of  the  devices.  Discussion  of  the  test 
plan  occurred  at  JWG  meetings  over  the  subsequent  seven  months,  and  Honeywell 
submitted  a  final  test  plan  to  ARI  in  November  1981. 

$ 

In  the  meantime,  device  development  and  construction  were  underway. 

The  schools  provided  the  contractors  with  lesson  plans  and  course  material 
which  was  to  be  incorporated  into  the  devices.  Contractor  representatives 
spent  time  at  both  schools,  meeting  with  instructors  and  subject  matter 
experts.  However,  construction  of  both  devices  fell  behind  the  original 
schedule.  Contractors  report  that  slippages  were  due  to  the  extensive 
changes  required  following  the  changes  in  task  selection  after  front  end 
analyses  and  after  contract  award. 

In  September  1981,  one  year  after  the  Phase  II  awards,  the  JWG  conducted 
In-Plant  Reviews  of  both  devices.  The  Sevi lle/Burtek  device  was  not  assembled 
nor  had  courseware  been  developed,  but  the  companies  projected  a  January  1982 
delivery  date  (three  months  behind  schedule).  Grumman  demonstrated  hardware, 
software,  and  courseware,  but  also  anticipated  late  deliveries.  They  pro¬ 
jected  a  delivery  to  USAOC&S  in  February  1982  and  one  to  USAADS  in  April  1982. 
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It  was  thought  at  the  time  of  the  In-Plant  Reviews  that  both  device 
deliveries  might  be  even  later  than  expected,  and  that  the  device  evalua¬ 
tion  contract  (SIMTRAIN  I)  might  actually  end  before  the  devices  could  be 
evaluated.  Honeywell  submitted  its  final  test  plan  in  November  1981,  and 
SAI,  as  subcontractor  to  Honeywell,  continued  to  plan  for  the  device 
evaluation.  Device  delays  persisted,  and  in  mid-1983,  ARI  awarded  a 
contract  (SIMTRAIN  I  Extension)  to  SAI  to  collect  and  analyze  data,  and 
to  submit  the  evaluation  report.  Honeywell  completed  its  work  on  the 
SIMTRAIN  I  project  and  is  no  longer  involved  in  the  AMTESS  effort. 

Problems  began  developing  at  the  other  test  site,  USAADS,  in  October 
1981,  nearly  a  year  before  any  device  would  be  delivered  there.  Conven-  ' 
tional  training  changed  from  a  self-paced  to  a  lock-step  mode  and  the 
school  was  unwilling  to  provide  self-paced  conventional  training  to  subjects 
in  the  evaluation.  The  primary  experimental  comparison  in  the  device  evalua¬ 
tion  was  that  of  conventional  to  simulator  training.  Now  the  validity  of 
that  comparison  was  threatened.  Again,  as  when  major  curriculum  changes 
occurred  at  USAOC&S,  perhaps  the  process  should  have  been  halted  and  a  new 
school  chosen  which  could  provide  conventional  training  that  met  evaluation 
requi rements . 

In  February  1982,  the  JWG  conducted  the  In-Plant  Acceptance  of  the 
Seville/Burtek  device.  According  to  an  ARI  trip  report,  the  acceptance 
involved  a  demonstration  of  the  missile  configuration  and  visual  inspection 
(but  no  demonstration)  of  the  automotive  configuration.  The  fact  that  a 
mismatch  existed  between  tasks  actually  taught^ by  the  devices  and  tasks 
expected  was  not  uncovered  at  that  time. 

Also  in  February  1982,  expecting  device  deliveries  in  the  near  future, 
SAI  hired  and  trained  its  data  collectors.  During  the  next  several  months, 
SAI  continued  to  develop  performance  tests  and  prepare  for  the  evaluation. 

ARI  restricted  SAI's  access  to  the  contractors  so  that  the  evaluation  would 
be  conducted  by  a  totally  independent  party. 

In  April  1982,  the  Seville/Burtek  device  was  delivered  to  USAOC&S. 

When  it  arrived,  it  was  discovered  that  the  tasks  for  which  the  perform¬ 
ance  tests  had  been  developed  were  not  included  and  that  the  tasks  taught 


by  the  device  were  actually  from  the  63H  MOS  originally  planned.  Not 
only  were  the  tasks  taught  by  the  device  a  surprise,  the  tasks  pertained 
to  a  different  type  engine  than  was  expected.  At  the  last  minute,  SAI 
developed  new  performance  tests  to  match  the  device,  and  the  evaluation 
began  in  early  May  1982.  Sevi lle/Burtek  was  not  pleased  with  the  evalua¬ 
tion  plans  and  protested  because  not  all  the  tasks  taught  by  their  device 
were  included  in  the  evaluation.  In  response  to  the  protest,  the  per¬ 
formance  tests  were  revised  and  a  new  criterion  was  added.  The  new 
criterion  was  the  extent  to  which  subject  matter  experts  felt  the  device 
covered  the  required  program  of  instruction. 

The  final  In-Plant  Acceptance  of  the  Grumman  device  occurred  in 
July  1982,  and  was  accepted  at  USAOC&S  in  August  1982. 


In  September  1982,  Seville/Burtek  delivered  its  device  to  USAADS. 

This  device  matched  what  was  expected,  and  the  evaluation  at  USAADS  began 
in  October  1982. 

The  Grumman  device  was  delivered  to  USAADS  in  November  1982.  Again, 
there  were  problems.  Performance  tests  for  some  of  the  skills  taught  by 
the  device  could  not  be  implemented  because  the  skills  were  too  dangerous 
to  perform  on  actual  equipment  because  of  high  voltage.  Therefore,  testing 
occurred  on  the  simulator.  Further,  use  of  the  device  was  clearly  supple¬ 
mental  to  conventional  training.  This  fact,  coupled  with  the  problem 
created  by  the  change  in  conventional  training  from  self-paced  to  lock-step 
prevented  a  comparison  of  control  and  experimental  training. 

More  problems  arose  at  USAADS  concerning  the  evaluation  of  the  Grumman 
device.  The  experimental  plan  called  for  10  students  each  in  the  simulator 
and  conventional  training  groups.  The  school,  however,  did  not  make  24C10 
(entry-level)  students  available  for  the  evaluation,  apparently  because 
their  participation  in  the  study  would  interfere  with  their  training  in 


mechanics  which  was  the  school’s  responsibility  to  provide.  The  school 
instead  provided  10  advanced  students  from  other  MOSs  for  simulator  training 
and  provided  no  students  from  conventional  training  for  testing.  Thus,  no 
experimental  comparison  between  the  Grurmnan  device  and  conventional  training 
was  conducted  at  USAADS. 

SAI  completed  the  evaluation  of  both  devices  at  both  schools  by  mid- 
June  1983.  The  complete  report  on  the  device  evaluation  is  in  progress. 

4.  Summary.  During  Phase  I,  PM  TRADE,  the  schools,  the  advisory 
group,  and  the  contractors  worked  together  well  enough  that  the  four 
contractors  conducted  front  end  analysis,  designed  devices,  and  delivered 
final  reports  in  a  timely  manner.  Several  problems  arose  during  Phase  I, 
however,  which  were  unresolved  and  carried  over  into  Phase  II.  First,  no 
memorandum  of  understanding  bound  PM  TRADE,  ARI,  the  contractors  and  schools. 
Second,  government  monitoring  of  the  frequent  contractor  and  school  contacts 
either  did  not  occur  or  is  not  documented.  This  practice  continued  on  into 
Phase  II  with  schools  and  contractors  meeting  but  without  close 
monitoring.  Third,  the  goal  of  the  schools  was  to  obtain  better  main¬ 
tenance  training  programs,  and  the  schools  apparently  did  not  officially 
agree  to  cooperate  fully  with  device  research.  Finally,  ARI  did  not 
assume  any  special  role  during  Phase  I,  yet  ARI  would  administer  the 
device  evaluation  projects. 

Phase  II  and  the  device  evaluation  seemed  to  be  characterized  by  mis¬ 
understanding.  Even  before  the  Phase  II  contracts  were  awarded,  changes  in 
device  specifications  threatened  problems.  The  first  changes  in  planning 
for  task  selection  for  the  devices  were  required  in  July  1980.  In  spite 
of  all  the  meetings,  phone  calls,  and  trips  between  schools  and  contractors 
that  occurred  over  a  21-month  period,  the  first  device  delivered  (and  sub¬ 
sequently  two  more  device  deliveries)  did  not  teach  the  tasks  many  agencies 
thought  it  should.  Unexpected  changes  in  both  the  schools'  conventional 
training  programs  threatened  the  validity  of  the  device  evaluation.  The 
device  evaluation  was  conducted,  but  the  evaluation  was  able  to  address 
only  a  portion  of  the  questions  it  sought  to  answer. 
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C.  Outcomes  of  the  AMTESS  Experience 

In  section  I.B.  above,  a  chronology  of  AMTESS  project  events  was 
presented.  In  this  section,  a  summary  of  key  participants'  opinions 
about  the  AMTESS  process  is  presented.  All  opinions  came  from  structured 
interviews  described  in  sections  I. A. 2.  and  I. A. 3.  above.  The  opinions 
presented  here  do  not  represent  official  views  of  organizations. 

This  discussion  is  organized  around  five  topics: 

1.  Objectives  of  AMTESS 

2.  Performance  of  organizations  involved 

3.  Coordination  of  effort 

4.  The  device  evaluation 

5.  The  future  of  AMTESS 

7.  Objectives  of  AMTESS.  Table  2  presents  a  sumary  of  opinions 
of  the  nine  organizations  concerning  the  primary  objectives  of  AMTESS. 

The  data  in  Table  2  suggest  that  there  was  good  agreement  among  organiza¬ 
tions  that  the  primary  AMTESS  objective  was  to  develop  concepts,  then  using 
the  concepts,  produce  a  breadboard  (or  forerunner)  generic  maintenance 
trainer.  The  trainer  was  to  include  a  common  core  and  various  3D  modules. 

Some  organizations  (PM  TRADE,  ARI,  and  SAI,  for  example)  added  that 
the  development  of  front  end  analysis  procedures  that  might  apply  to  the 
development  of  other  training  simulators  was  a  second  AMTESS  objective. 

Most  organizations,  however,  did  not  mention  this  objective. 

In  spite  of  general  agreement  about  the  primary  AMTESS  objective, 
representatives  from  seven  of  the  nine  organizations  stated  that  the 
objectives  and  process  for  AMTESS  were  unlike  other  device  development 
processes  because  of  the  lack  of  specificity  received  from  the  Army. 
According  to  interviewees,  the  development  process  for  most  devices  is 
usually  more  clearly  spelled  out  than  it  was  for  AMTESS  devices.  The 
AMTESS  objective  was  very  general. 
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TABLE  2.  OPINIONS  OF  ORGANIZATIONS'  REPRESENTATIVES  CONCERNING  AMTESS 
OBJECTIVES 


ORGANIZATION  PRIMARY  AMTESS  OBJECTIVE  HAS  AMTESS  PROCESS 

REPRESENTED  DIFFERED  FROM  OTHER 

DEVICE  PROCUREMENT 
PROCESSES? 


PM  TRADE 

Develop  generic  maintenance  trainer 
that  allows  for  broad  spectrum  of 
training  and  transfer  to  any  main¬ 
tenance  training  area. 

Same  as  other  6.2 
programs . 

ARI 

Develop  general  model  of  a  generic 
maintenance  trainer  which  includes 
a  core  module  and  MOS-specific  3D 
modules. 

Yes;  this  was  R&D. 

USAOC  &S 

To  challenge  industry  to  develop 
a  generic  maintenance  trainer 
involving  2D  and  3D  modules. 

Yes;  Army  is  usually 
more  specific  in  its 
desires. 

USAADS 

Develop  maintenance  training  con¬ 
cepts  and  then  develop  appropriate 
hardware. 

Yes;  this  was  R&D. 

ATSC 

Develop  a  generic  way  to  train 
maintenance. 

Yes;  this  was  R&D. 

GRUMMAN 

Develop  conceptual  design  and  then 
the  hardware  to  be  a  forerunner  of 
a  universal  maintenance  training 
device. 

Yes;  much  was 
unspecified. 

SEVILLE 

Develop  family  of  hands-on  mainte¬ 
nance  trainers. 

This  was  like  any 

R&D  effort. 

BURTEK 

Verify  cost  and  training  effective¬ 
ness  of  a  maintenance  simulator 
which  includes  common  core  hardware 
and  3D  modules  applicable  to  many 

MOSs,  especially  "low-end"  MOSs. 

Yes;  usually 
receive  more 
specifications. 

SAI 


Design  and  build  modular,  recon 
figurable  low-cost  breadboard 
generic  maintenance  trainer. 


Yes;  this  device 
was  generic. 


2.  Organization  Performance.  This  section  includes  a  discussion  of 
participants'  comments  about  the  performance  of  the  organizations  involved 
in  AMTESS.  Interviewees  commented  on  organizations'  performance  in  general, 
and  with  respect  to  pertinent  critical  tasks.  The  goal  of  this  discussion 
is  to  present  general  themes,  not  to  specifically  identify  the  author  of 
each  comnent  discussed. 

a.  PM  TRADE.  Several  interviewees  felt  that  PM  TRADE  had  done  a 
good  job  developing  the  AMTESS  concept;  however,  most  participants  felt 
that  the  statements  of  work  (SOWs)  pertaining  to  the  concept  were  too 
open-ended  with  too  much  interpretation  left  to  the  contractors.  Only  a 
few  interviewees  commented  on  PM  TRADE'S  proposal  review  process,  but 
apparently  the  majority  felt  that  a  standard  review  process  was  followed. 
Most  felt  that  PM  TRADE  should  have  been  more  of  a  presence  in  monitoring 
in-plant  development  of  both  devices.  Similarly,  the  interviewees  felt 
that  PM  TRADE  should  have  been  more  active  in  monitoring  the  device  evalua¬ 
tion.  Several  interviewees  commented  that  PM  TRADE  should  have  provided 
more  leadership  in  general  to  the  AMTESS  project.  However,  PM  TRADE  was 
generally  pleased  with  its  performance. 

b.  AfU.  Most  interviewees  did  not  comment  on  ARI's  performance  in 
the  proposal  review  process;  standard  procedures  were  followed.  One  inter¬ 
viewee  felt  the  device  evaluation  award  to  Honeywell  was  "improper"  because 
Honeywell  was  a  potential  bidder  in  future  AMTESS  procurements.^  With 
respect  to  ARI's  SOW  for  the  device  evaluation,  two  interviewees  felt  it 
was  vague;  one  of  those  interviewees,  however,  felt  the  SOW  was  acceptable. 

Representatives  of  all  organizations  which  commented  felt  that  ARI's 
in-plant  monitoring  of  device  development  was  good.  ARI  participated  both 
In  In-Plant  Reviews  and  Joint  Working  Group  meetings. 

'Exclusion  of  bidders  Is  not  ARI's  prerogative.  ARI  acts  as  an  agent  for 
the  Defense  Supply  Service  in  contract  awards. 


Most  interviewees  felt  that  ARI's  test  plan  was  too  general,  but  many 
felt  that  ARI  did  a  good  job  monitoring  the  device  evaluations.  AR1 
representatives  felt  that  limited  funds  for  travel  prevented  ARI  personnel 
from  visiting  Ft.  Bliss  as  often  as  they  would  have  liked. 

One  interviewee  remarked  that  ARI's  approach  to  AMTESS  was  too 
academic  and  therefore  impractical.  Representatives  from  two  organizations 
felt  that  a  stronger  relation  between  ARI  and  PM  TRADE  would  have  helped 
the  project. 

c.  Schools.  Opinions  were  about  evenly  divided  on  the  schools' 
work  wi'th  the  contractors  during  the  front  end  analyses.  About  half  said 
the  schools  could  have  worked  more  closely  with  the  contractors  and  about 
half  felt  the  schools  and  contractors  worked  adequately  with  each  other. 

In  general,  interviewees  felt  that  the  schools',  or  the  subject  matter 
experts',  roles  in  both  the  in-plant  monitoring  of  device  development  and 
device  acceptance  had  not  been  clearly  specified  by  the  Joint  Working  Group. 
However,  most  individuals  felt  that  the  schools  did  well  with  device  develop¬ 
ment  monitoring.  Most  interviewees  also  felt  that  a  much  stronger  relation 
between  subject  matter  experts  and  the  schools  would  have  improved  device 
development. 

Opinions  were  mixed  regarding  the  schools'  cooperation  with  the  device 
evaluation.  Generally,  the  schools  cooperated  as  best  they  could,  but  the 
evaluation  lacked  support  from  top  school  administrators.  Therefore,  no 
one  deeply  involved  in  AMTESS  had  sufficient  authority  over  school  personnel 
(instructors,  subject  matter  experts,  course  writers,  for  example)  to  get 
the  cooperation  required  for  a  fully  successful  evaluation.  The  schools’ 
intentions  were  good,  but  there  was  no  official  cooperative  agreement. 

d.  Contractors.  Both  contractors  were  pleased  with  their  front  end 
analyses,  preliminary  systems  engineering  design,  and  life  cycle  cost  estimates. 
Most  interviewees  did  not  comment  on  those  aspects  of  contractor  performance. 


In  general,  most  interviewees,  especially  the  contractors,  were  satisfied 
with  the  devices.  According  to  some  device  contractor  representatives,  the 
contractors  received  little  direction  from  the  Army,  were  faced  with  mid¬ 
stream  changes  in  device  requirements,  yet  both  contractors  produced  working 
hardware  and  software.  However,  one  spokesman  was  disappointed  in  both  devices 
because  they  were  not  stimulating  and  were  cumbersome;  representati ves  from  two 
organizations  pointed  out  the  problems  caused  by  late  device  deliveries; 
several  interviewees  mentioned  major  problems  caused  by  disrepair. 

Both  contractors  were  satisfied  with  their  device  maintenance  service 
during  the  evaluation.  Grumman  provided  telephone  consultation  and  Seville/ 
Burtek  provided  a  technician  on-site.  Although  both  contractors  employed  a 
maintenance  plan,  a  clear  majority  of  interviewees  felt  the  on-site  technician 
provided  better  service. 

e.  SAI.  Most  organizations  felt  that,  under  the  difficult  circumstances, 
SAI  developed  satisfactory  performance  tests  for  the  device  evaluation.  While 
ARI  was  pleased  with  SAI's  test  plans,  several  representatives  did  not  see 
copies  of  the  test  plan  as  it  was  revised.  Thus,  several  interviewees  did  not 
know  that  test  plans  existed. 

As  for  the  qualitative  and  quantitative  data  gathered  by  SAI  in  the 
device  evaluation,  some  individuals  did  not  comment  because  the  evaluation 
report  is  not  written.  Most  interviewees  were  pleased  with  data  collection 
saying  SAI  did  well  given  the  constraints.  The  strong  efforts  of  SAI's 
two  on-site  data  collectors  were  praised.  The  TRAINVICE  evaluation  (a 
portion  of  device  evaluation  involving  user  ratings  of  device  features), 
while  being  relevant  and  appropriate,  was  criticized  by  both  schools  as 
being  too  long. 

One  organization  commented  that  SAI  did  a  good  job  and  was  flexible  in 
their  planning.  SAI  was  pleased  with  its  own  performance. 

f.  ATSC.  ATSC,  as  a  member  of  the  Joint  Working  Group,  was  to  represent 
TRADOC's  perspective.  Nearly  all  individuals  commenting  on  ATSC  felt  that 
ATSC  representatives  took  an  active  role  in  AMTESS. 


3.  Coordination  of  Effort.  Interviewees  were  asked  to  identify 
the  locus  of  responsibility  for  coordination  of  AMTESS  efforts  and  to 
characterize  the  coordination  between  organizations. 

First,  all  interviewees  stated  that  PM  TRADE  was  responsible  for 
managing  AMTESS.  Individuals  were  mixed  in  opinions  concerning  the 
adequacy  of  PM  TRADE'S  authority  and  resources  to  manage  AMTESS. 

Representatives  from  about  half  of  the  organizations  felt  that  PM  TRADE 
had  inadequate  authority  and/or  resources  to  manage  AMTESS.  However,  many 
representatives  felt  that  PM  TRADE  had  sufficient  authority  and  resources. 

Several  interviewees  felt  that  limited  resources  and/or  authority 
presented  problems  for  PM  TRADE'S  management  effort.  However,  representa¬ 
tives  of  two  organizations  felt  that  PM  TRADE'S  management  was  satisfactory. 

PM  TRADE  was  satisfied  with  its  own  participation  in  AMTESS. 

Related  to  PM  TRADE'S  responsibility  is  the  responsibility  of  the 
Joint  Working  Group  (JWG)  and  the  effectiveness  of  JWG  meetings  in  coordinating 
AMTESS  activities.  Several  interviewees  said  that  the  JWG  did  not  provide 
clear  directions  to  the  organizations  involved.  Rules,  meeting  objectives, 
and  minutes  might  have  strengthened  the  JWG's  effectiveness  in  the  opinion 
of  several  interviewees.  One  interviewee  commented  that  the  JWG  was  stronger 
early  in  AMTESS  than  in  the  later  time  period.  One  representative  felt  that 
the  JWG  worked  well . 

Problems  in  coordination  of  activities  among  the  organizations  involved 
were  the  result  of  lack  of  direction  from  the  JWG  according  to  a  few  inter¬ 
viewees.  Two  interviewees  viewed  coordination  as  satisfactory. 

Interviewees  were  asked  to  characterize  the  coordination  between  their 
organization  and  each  other  organization.  Most  interviewees  felt  that  their 
organization  had  contact  with  PM  TRADE  only  through  JWG  meetings;  thus,  most 
felt  that  coordination  with  PM  TRADE  was  not  as  good  as  it  should  have  been. 
Some  individuals  pointed  out  that  PM  TRADE  was  always  available  by  phone. 


The  coordination  between  ARI  and  SAI  was  judged  to  be  satisfactory  by 
representatives  of  both  organizations.  The  schools  felt  that  a  stronger 
relation  between  themselves  and  ARI  would  have  been  helpful  although  both 
schools  had  frequent  telephone  contact  with  ARI 

Most  interviewees  commenting  felt  that  coordination  between  their 
organization  and  the  schools  was  satisfactory.  Representatives  from 
one  organization  commented  that  coordination  was  better  with  subject  matter 
experts  than  with  top  school  administrators. 

Most  interviewees  felt  that  coordination  between  their  organization 
and  the  contractors  was  minimal  and  that  more  contact  was  needed.  Where 
contacts  were  frequent  between  the  contractors  and  schools,  apparently 
monitoring  of  the  contacts  was  needed.  In  general,  participants  felt  that 
the  problems  surrounding  device  delivery  and  evaluation  might  have  been 
avoided  had  coordination  and  leadership  been  more  apparent. 

Coordination  with  SAI  was  satisfactory  according  to  representatives 
of  several  organizations.  However,  one  representative  of  USAADS  felt  that 
more  frequent  on-site  contact  with  a  senior  SAI  staff  member  would  have 
helped.  From  SAI's  point  of  view,  cost  prohibited  more  frequent  travel  to 
USAADS. 

All  individuals  commenting  felt  that  coordination  with  ATSC  at  the 
JWG  meetings  was  satisfactory. 

4.  Device  Evaluation.  It  is  not  the  purpose  of  this  report  to 
summarize  the  results  of  the  device  evaluation.  The  report  presenting 
the  results  of  the  empirical  device  evaluation  is  separate  from  this  report. 
This  section  contains  opinions  about  the  evaluation  (which  was  still  in 
progress  at  the  time  of  the  interviews). 

Interviewees  were  asked  to  comment  on  several  aspects  of  device  evalua¬ 
tion.  Among  those  aspects  were  selection  of  test  tasks  and  the  comprehensive¬ 
ness  and  validity  of  the  evaluation. 


Most  representatives  felt  that  ARI  was  responsible  for  asking  the 
schools  to  select  the  tasks  used  in  device  evaluation.  Many  interviewees 
did  not  know  who  selected  tasks  to  be  taught.  A  few  thought  the  schools 
(with  contractor  input)  actually  selected  the  tasks  to  be  taught  by  the 
devices. 

Most  interviewees  felt  that  the  validity  of  the  evaluation  has  been 
compromised.  Two  important  reasons  mentioned  were  too  few  tasks  tested 
and  an  insufficient  number  of  students.  However,  one  interviewee  remarked 
that  the  problems  encountered  are  typical  in  field  studies. 

5.  The  Future  of  AMTESS.  Interviewees  were  asked  if  they  felt  that 
the  AMTESS  concept,  embodied  in  the  prototypes,  has  lived  up  to  expecta¬ 
tions.  They  were  also  asked  if  the  AMTESS  concept  is  worth  pursuing. 

Opinions  were  mixed  concerning  the  prototypes'  representation  of  the 
AMTESS  concept.  Most  interviewees,  especially  contractor  representatives, 
felt  that  the  devices  had  "proven"  the  concept  and/or  surpassed  expectations 
Those  people  who  felt  the  devices  did  not  live  up  to  expectations  most  often 
mentioned  operational  problems  with  the  equipment. 

Agreement  was  widespread  that  the  AMTESS  concept  is  worth  pursuing. 
Individuals  varied  widely,  however,  in  their  opinions  of  how  AMTESS  should 
be  pursued.  Most  felt  that  more  research  is  needed.  Several  felt  that  the 
3D  modules  should  be  designed  to  reflect  general  maintenance  tasks,  not 
specific  (and  frequently  changing)  MOSs.  Some  contractor  interviewees 
offered  some  questions  needing  answers.  For  example,  is  the  sophisticated 
integration  of  2D  and  3D  components  necessary  to  teach  very  simple  skills? 
How  can  one  core  component  be  designed  to  support  many  different  3D  modules? 
Is  remove/replace  capability  as  important  as  troubleshooting  capability? 

Are  the  simulators  supposed  to  replace  or  supplement  existing  training? 

The  tone  of  interviewee  comments  about  the  future  of  AMTESS  was  hopeful 
In  spite  of  problems  encountered  during  AMTESS,  the  need  for  tested  AMTESS 
devices  is  clear  to  all  the  participants. 


D.  Summary 


The  preceding  sections  have  presented  a  brief  history  of  the  AMTESS 
project  and  participants'  opinions  about  critical  events  in  the  AMTESS 
project.  Perhaps  the  largest  problem,  which  arose  during  Phase  I  and 
carried  over  into  Phase  II,  was  the  absence  of  a  memorandum  of  understanding. 
Many  organizations  did  not  appreciate  their  specific  responsibilities  to  the 
program,  and  gaps  in  performance  were  evident.  Progress  made  during  Phase  I 
might  be  attributed  to  agreement  about  general  project  goals  and  adequate 
cooperation  between  schools  and  contractors.  Phase  I  final  reports  were 
submitted  in  a  timely  fashion. 

During  Phase  II,  coordination  and  communication  suffered  as  the  number 
of  participants  increased  and  the  interfaces  between  participants  became 
more  complex.  Two  simultaneous  efforts,  device  construction  and  device 
evaluation,  should  have  been  more  interactive  than  they  were.  However, 
interaction  was  insufficient,  and  as  a  result,  the  devices  did  not  meet 
all  the  schools'  expectations. 

In  spite  of  the  problems  encountered,  participants  in  the  AMTESS  project 
felt  that  many  project  goals  were  accomplished.  Participants  shared  the 
idea  that  generic  maintenance  training  simulators  will  be  important  in  Army 
training.  Most  also  felt,  however,  that  the  devices  developed  during  AMTESS 
need  considerable  improvement  before  they  are  ready  for  classroom  use. 


II.  AMTESS  DEVICE  FEATURES 


This  section  presents  interviews  opinions  and  expert  judgments  about 
the  features  of  the  device.  First,  the  procedures  used  to  collect  opinions 
and  judgments  are  described.  Next,  a  summary  of  opinions  is  presented, 
first  the  Sevil le/Burtek  device,  then  for  the  Grumman  device. 

A.  Procedures 

The  data  for  this  section  come  from  two  sources:  structured  inter¬ 
views  and  device  feature  ratings  from  a  small  panel  of  experts. 

1.  Structured  Interviews.  Opinions  about  device  features  were 
obtained  in  the  structured  interviews  mentioned  in  section  I. A. 2.  above. 

The  reader  is  referred  to  that  section  for  interview  procedures. 

All  interviewees  were  asked  to  conment  on  the  concept  and  implementa¬ 
tion  of  specific  device  features.  They  were  also  queried  about  the  overall 
functioning  of  the  devices.  Contractors  were  also  asked  to  comment  briefly 
on  the  rationale  behind  device  features. 

2.  Expert  Ratings.  Quantitative  ratings  of  device  features  were 
obtained  to  augment  the  qualitative  data  collected  in  the  structured 
interviews.  Ratings  were  obtained  from  four  experts,  all  unbiased  and 
knowledgeable  about  both  devices.  Three  SAI  and  one  ARI  personnel  rated 
the  devices.  One  SAI  and  one  ARI  personnel  rated  features  of  both  devices 
used  at  both  USAADS  and  USAOC&S.  Another  SAI  employee  rated  features  of 
both  devices  used  at  USAADS  and  a  third  SAI  employee  rated  features  of 
both  devices  used  at  USAOC&S.  Thus,  three  sets  of  ratings  were  obtained 
for  both  devices  at  each  school. 

The  training  device  features  evaluation  form  given  to  each  judge  was 
designed  such  that  the  ratings  could  be  easily  subjected  to  a  multi -attribute 
utilities  analysis  (Swezey,  1979).  Multi-attribute  utilities  analysis  is 
an  applied  technique  useful  to  combine  quantitative  subjective  data  about 
several  different  attributes  from  several  judges.  This  technique  is 


particularly  applicable  when  attributes  may  be  evaluated  on  different 
dimensions  of  value.  The  ratings  for  each  attribute  are  weighted  according 
to  a  constant  derived  from  each  dimension  of  value.  Then  the  weighted 
ratings  for  each  attribute  are  summed.  The  sum  for  each  attribute  represents 
its  value  with  respect  to  all  the  dimensions.  Using  these  sums,  attributes 
may  be  rank  ordered. 

For  example,  each  dimension  of  value  is  assigned  a  weight  based  on  its 
importance,  in  the  judges'  opinions,  relative  to  all  other  dimensions.  Each 
judge  rates  each  attribute  along  each  dimension.  To  illustrate,  an  attribute 
might  receive  a  20  on  one  dimension  and  an  80  on  another.  The  20  and  80  are 
multiplied  by  the  weights  for  their  respective  dimensions.  Then,  those  two 
numbers  are  summed.  The  sums  for  all  judges  are  summed,  then  the  attributes 
rank  ordered.  This  procedure  is  explained  in  detail  below. 

The  ratings  form  used  contained  three  sections.  A  copy  of  the  form 
appears  in  Appendix  C.  In  the  first  section,  judges  were  asked  to  rank 
four  dimensions  of  value  in  order  of  their  importance  as  criteria  for 
features  in  a  generic  maintenance  training  simulator.  Judges  were  instructed 
to  rank  the  least  important  dimension  "10,"  then  rank  the  other  dimensions 
in  relation  to  the  least,  important  dimension  and  in  relation  to  each  other. 
For  example,  a  dimension  rated  "30"  is  three  times  more  important  than  the 
dimension  ranked  "10"  and  half  as  important  as  a  dimension  ranked  "60." 

Each  judge's  ranks  summed  to  100. 

The  four  dimensions  and  their  descriptions  are  listed  below: 

Concept  -  Is  the  concept  behind  the  feature  important  in 
a  generic  maintenance  training  simulator?  (For 
example,  the  concept  behind  both  a  video  disc  and 
a  slide  projector  may  be  the.  same,  that  of  auto¬ 
matic  presentation  of  visual  material.  The  con¬ 
cept  dimension  asks  if  the  concept  is  important.) 

Implementation  -  Does  the  feature  truly  embody  and  express 
the  concept  behind  it? 

Operation  -  Is  the  feature  easy  to  use  and  in  good  working 
order? 


Motivation  -  Is  the  feature  appealing  and  does  it  capture 
interest  in  learning? 


In  the  second  section,  judges  were  asked  to  rate  ten  Grumman  device 
features  along  each  of  the  four  dimensions.  Judges  were  asked  to  consider 
a  typical  trainee  and  the  MOSs  taught.  Judges  were  asked  to  use  a  scale 
of  0  to  100,  with  0  minimal  and  100  maximal.  Ratings  were  orthogonal  and 
did  not  have  to  sum  to  100.  Grumman  device  features  at  USA0C&S  were 
rated  separately  from  the  Grumman  device  at  USAADS.  The  ten  features 
rated  were: 

•  3D  module 

•  Student  performance  record 

•  Instructor  CRT 

•  Video  disc 

•  Touch  panel ,  student  CRT 

•  Editing  system 

•  Request  help 

•  Repeat  lesson  option 

•  Mandatory  instructor  call  after  two  errors 

•  Performance  feedback 


In  section  three,  judges  rated  13  features  of  the  Seville/Burtek 
device  using  the  same  procedures  as  for  the  Grumman  ratings  described 
above.  The  13  features  are: 

•  3D  module 

e  Student  performance  record 

•  Instructor  CRT 

•  Slide  projector  unit 
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Student  response  panel 
Editing  system 
Student  CRT 

Instructor  control  panel 
Remove/replace  capability 
Random  malfunction  selection 
Performance  feedback 
Troubleshoot  only  mode 
Sound  effects 


The  first  step  in  data  analysis  was  assigning  a  weight  to  each  of  the 
four  dimensions  of  value.  This  was  done  by  summing  the  ranks  of  all  judges 
on  each  of  the  four  dimensions  of  value.  The  sum  for  each  dimension  was 
divided  by  the  sum  of  the  sums  (400)  and  multiplied  by  100.  These  numbers 
were  used  as  the  dimension  weights.  (Raw  data  for  each  judge  are  presented 
in  Appendix  D.) 

According  to  our  experts,  the  most  important  dimensions  in  evaluating 
features  in  generic  maintenance  training  simulators  are,  in  order,  concept, 
operations,  implementation,  and  motivation.  The  weights  for  these  dimensions 
are  32.5,  28.8,  23.8,  and  15.0,  respectively.  These  weights  suggest  that 
concept  and  operations  are  highly  valued  and  close  in  value,  and  motivation 
is  clearly  the  least  important  criterion. 

Next,  each  judge's  ratings  for  each  feature  on  each  dimension  were 
weighted.  Then,  the  four  weighted  ratings  (one  of  each  dimension)  for 
each  feature  were  summed.  Finally,  ratings  for  each  feature  were  summed 
across  judges  (or  across  three  sets  of  ratings).  As  mentioned  above,  two 
experts  rated  both  devices  at  both  schools.  A  third  and  fourth  expert  rated 
devices  used  at  USAAOS  and  USA0C&S,  respectively.  Thus,  three  complete  sets 
of  ratings  were  available.  The  third  set  combines  judgments  from  the  third 
and  fourth  experts.  The  grand  sum  obtained  represents  feature  value  taking 
all  four  dimensions  of  value  into  account. 


The  results  of  the  multi -attribute  utilities  analysis  are  presented 
below  in  section  II. B. 2.  for  the  Sevi  1  le/Burtek  device  and  section  II. C. 2. 
for  the  Grumman  device. 

B.  Results:  The  Sevi 1 le/Burtek  Device 

1.  Opinions  about  Features.  Information  in  this  section  comes  from 
structured  interviews.  The  interview  protocol  contained  13  device  features 
for  interviewees  to  comment  on.  (It  was  considered  inappropriate  for  device 
contractors  to  comment  on  each  other's  devices.)  In  addition,  the  protocol 
contained  general  questions  about  feature  strengths  and  weaknesses. 

a.  3D  module.  Almost  all  interviewees  thought  the  3D  module  was  good, 
even  excellent.  The  module  was  praised  for  its  high  fidelity,  simplicity, 
and  good  operational -record. 

There  was  an  assortment  of  criticisms.  One  interviewee  commented  that 
the  module  at  USAOC&S  was  too  specific,  loud,  and  intimidating.  One  inter¬ 
viewee  mentioned  that  many  tasks  are  much  easier  to  perform  on  the  module 
than  on  the  real  equipment.  That  interviewee  recommended  increased  fidelity 
in  the  3D  module  to  include  improvements  with  the  fuel  line,  the  oil  line  and 
fittings,  and  addition  of  an  oil  dipstick.  Further,  an  increase  in  difficulty 
of  removing  the  starter  and  using  the  STE/ICE  device  was  recommended.  This 
interviewee  also  suggested  that  some  tool  usage  skills  were  not  of  high 
enough  fidelity.  For  example,  bolts  could  not  be  torqued  to  specification 
because  of  risk  of  damage  to  the  simulator. 

Overall,  representatives  of  all  organizations  commented  that  the  concept 
behind  the  3D  module  was  good  and  implementation  was  successful.  Several 
felt  the  realistic  module  was  motivating  to  students. 

b.  Student  performance  record.  This  feature  received  unifc< . i 


praise  for  its  completeness  and  dependability.  Seville  commented  that  the 
design  of  the  record  was  based  on  the  task  analysis  performed  in  the  front 
end  analysis. 
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A  few  organizations  felt  that  some  codes  used  on  the  record  were 
difficult  to  interpret.  Another  criticism  concerned  a  software  problem. 
Occasionally,  while  working  on  the  3D  module,  the  student  would  correctly 
push  or  pull  on  parts  to  repair  them.  However,  this  motion  caused  errors 
to  register  and  these  "errors"  were  reflected  on  the  performance  record. 

c.  Text  presentation  on  student  CRT.  Text  was  presented  on  the 
student  CRT.  Text  was  praised  as  being  well  coordinated  with  the  3D 
module  and  technical  manuals.  Interviewees  both  praised  and  criticized 
the  simplicity  of  the  text  presented.  One  interviewee  noted  that  start-up 
safety  training  had  been  omitted  from  instruction;  this  omission  could 
cause  performance  problems  on  actual  equipment.  However,  the  concensus 
was  that  this  feature  worked  well. 

d.  Instructor  CRT.  This  CRT  presented  screens  to  help  the  instructor 
set  up  the  device.  It  also  displayed  student  progress  information. 
Representatives  from  all  organizations  found  this  feature  good,  even 
excellent.  In  its  completeness  and  dependability. 

e.  Slide  projector  unit.  Representatives  from  all  organizations 
thought  the  use  of  a  slide  projector  in  a  generic  maintenance  trainer  was  a 
good,  if  not  excellent,  feature.  Implementation  problems  occurred,  however. 
One  interviewee  felt  the  unit  should  be  abandoned  because  of  frequent  mal¬ 
functions.  Several  organizations  criticized  its  dependability  and  mentioned 
a  chronic  overheating  problem.  One  interviewee  mentioned  a  focusing  problem. 
One  interviewee  commented  that  more  photographs  were  needed  on  the  slides; 
many  slides  were  redundant  with  the  technical  manuals.  One  Interviewee 
mentioned  that  many  slides  appeared  backwards  on  the  screen.  However, 
another  participant  thought  that  material  from  the  manuals  on  a  slide  held 

a  student's  Interest  more  than  the  same  material  in  the  manual.  Overall, 
most  Interviewees  liked  the  flexibility  and  low  cost  of  the  slide  projector. 

f.  Student  response  panel .  Opinions  were  mixed  about  the  response 
panel.  Some  individuals  liked  It  more  than  a  keyboard  or  touch  panel;  some 
liked  it  less.  One  criticism  was  that  several  buttons  had  to  be  pressed  to 


enter  one  response,  thus  making  entries  complicated.  A  more  SDecific 
criticism  concerned  problems  with  the  service  button  on  Task  #18  in  the 
remove/ rep lace  thermostat  sequence.  Generally,  however,  most  interviewees 
liked  its  simplicity  and  dependability. 

g.  Editing  system.  The  editing  system  allowed  the  instructor  to 
change  the  instructional  material.  Participants  from  all  organizations 
rated  the  simple-to-use  editing  system  as  good  to  excellent  -  a  strong 
point  to  the  system. 

h.  Remove/replace  capability.  Most  interviewees  reported  that  this 
feature  was  often  in  need  of  repair,  but  that  the  concept  behind  the  feature 
was  good.  According  to  some  interviewees,  this  feature  was  useful  for 
beginning  level  students  more  so  than  for  advanced  students. 

i.  Random  malfunction  selection.  This  feature  sets  the  device  on 
a  randomly  selected  malfunction.  Interviewees  thought  this  was  a  good 
feature  although  it  was  not  used  often  at  USAADS.  One  participant  reported 
that  malfunction  selection  might  not  have  been  truly  random;  some  days  the 
same  malfunction  would  appear  repeatedly. 

j.  Performance  feedback.  The  feature  provided  feedback  to  the  student. 
Most  interviewees  felt  this  feature  was  adequate. 

k.  Troubleshoot  only  mode.  This  feature  gave  the  student  a  chance 
to  practice  troubleshooting.  Participants  thought  this  was  a  good,  if 
not  very  good,  feature.  One  individual  commented  that  this  feature  is 
especially  motivating  and  instructional  for  more  advanced  students. 

l.  Sound  effects.  Sound  effects  were  included  on  the  device  at 
USA0C4S.  Most  interviewees  thought  the  sound  effects  were  realistic  and 
good.  One  interviewee  thought  they  were  especially  motivating.  One 
interviewee  felt  the  sound  effects  were  not  important. 


in.  Instructor  contol  panel.  The  instructor  uses  this  panel  to  operate 
the  device.  Interviewees  felt  that  this  panel  was  a  good  idea,  straight¬ 
forward,  and  simple  to  use.  Seville  representatives  commented  that  this 
panel  was  designed  based  on  past  experience  and  on  the  task  analysis  com¬ 
pleted  in  the  front  end  analysis.  One  participant  commented  that  the  pre¬ 
session  system  check  feature  was  good. 

n.  Software.  Burtek  representatives  felt  that  their  software  programmed 
in  FORTRAN  IV  was  an  outstanding  feature  of  the  device.  It  was  highly  useful 
for  writing  programmed  instruction,  easy  to  troubleshoot  during  device  develop¬ 
ment,  and  compatible  with  military  requirements.  Further,  FORTRAN  IV  is  easy 
for  users  and  thus  users  might  not  be  dependent  on  Burtek  for  updates. 

Specific  benefits  of  the  programming  were  mentioned  by  some  interviewees. 
According  to  one,  the  capability  for  the  instructor  to  insert  a  malfunction 
was  a  powerful  feature.  In  fact,  this  feature  was  said  to  make  training  more 
authentic  than  actual  equipment  training  because  instructors  could  program 
malfunctions  they  could  never  insert  into  the  actual  equipment. 

2.  Expert  Ratings.  A  panel  of  experts  was  asked  to  rate  features  of 
the  Seville/Burtek  device  and  the  ratings  were  subjected  to  a  multi -attribute 
utilities  analysis.  Basically,  each  device  feature  was  rated  along  each  of 
four  dimensions:  concept,  implementation,  operations,  and  motivation.  Pro¬ 
cedures  used  for  rating  and  data  analysis  are  described  in  section  I I. A. 2.  above. 

Table  3  presents  each  judge's  weighted  rating  and  the  total  rating  for 
each  device  feature.  As  mentioned  in  section  II. A. 2.,  the  third  judge's 
ratings  are  a  combi  nation  of  the  ratings  from  the  expert  who  rated  the  USAADS 
devices  and  the  one  who  rated  the  USAOC&S  devices.  The  data  in  Table  3  are 
grand  totals  of  the  weighted  ratings  for  the  Seville/Burtek  device  at  USAOC&S 
and  USAADS.  Thus,  the  data  in  Table  3  suggest  how  valued  each  device  feature 
is,  taking  all  four  dimensions  into  account.  Features  in  Table  3  are  listed 
in  descending  order  by  rank. 


-35- 


r-_V 


(  i 


TABLE  3.  MULT I -ATTRIBUTE  UTILITIES  DATA,  WEIGHTED  RATINGS  FOR 
SEVILLE/BURTEK  DEVICE 


FEATURE 

J1 

J2 

J3 

0  Total 

1. 

Student  performance  record 

14,664 

18,200 

17,192 

50,056 

2. 

3D  module 

14,778 

18,349 

16,504 

49,631 

3. 

Troubleshoot  only  mode 

14,664 

16,324 

13,162 

44,150 

4. 

Performance  feedback 

12,231 

16,603 

14,462 

43,296 

5. 

Remove/replace  capability 

13,858 

15,760 

12,762 

42,380 

6. 

Student  CRT 

11,137 

15,373 

13,166 

39,676 

7. 

Random  malfunction  selection 

15,052 

14,829 

8,762 

38,643 

8. 

Student  response  panel 

12,764 

12,506 

9,540 

34,810 

9. 

Slide  projector  unit 

11,956 

11,036 

11,138 

34,130 

^Totavi 

Mean  *  41,863.6 
S.D  *  5,697.7 
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Table  3  shows  that,  according  to  our  experts,  the  student  performance 
record  and  the  3D  module  are  the  most  valued  features  in  the  Sevi 1 le/Burtek 
device.  The  total  value  for  these  features  exceeds  one  standard  deviation 
above  the  mean  of  this  distribution.  Table  3  also  shows  that  the  student 
response  panel  and  the  slide  projector  are  the  least  valuable  features  in 
the  Seville/Burtek  device.  The  total  values  for  those  two  features  fall 
more  than  one  standard  deviation  below  the  mean. 

In  a  very  general  way,  the  order  of  ranks  in  Table  3  parallels  the 
subjective  comments  made  by  personnel  who  gave  structured  interviews.  The 
student  performance  record  and  the  3D  module  received  high  praise.  The 
slide  projector  received  criticisms  for  operational  problems. 

Table  4  presents  the  multi -attribute  utilities  data  for  features  of 
the  Seville/Burtek  device,  separately,  by  school.  Although  judges'  ratings 
were  similar  for  the  device  at  each  school,  differences  were  seen.  Table  4 
shows  that  at  both  schools  the  student  performance  record  and  the  3D  module 
were  rated  the  most  valuable  device  features.  Values  for  these  features 
exceed  one  standard  deviation  above  the  mean.  However,  the  least  valuable 
feature  at.USAADS  was  the  slide  projection  unit,  and  the  response  panel  at 
USAOC&S.  Those  features  had  composite  ratings  less  than  one  standard  devia¬ 
tion  below  the  mean.  Nevertheless,  the  rank  order  of  features  on  this  device 
at  both  schools  is  very  similar. 

In  addition  to  the  features  mentioned  above,  four  device  features,  the 
instructor  CRT,  the  editing  system,  the  instructor  control  panel,  and  sound 
effects,  were  also  included  in  the  ratings  protocol.  However,  the  instructor 
CRT,  editing  system,  and  instructor  control  panel  were  not  included  in  the 
analysis  because  the  judges  felt  that  the  dimension  of  "motivation"  did  not 
apply  to  these  features.  Data  for  sound  effects  are  not  included  in  the 
analysis  because  sound  effects  were  used  only  in  the  device  at  USAOC&S.  Thus, 
data  for  these  four  features  are  incomplete. 

Weighted  ratings  in  a  multi -attribute  utilities  analysis  are  global  in 
the  sense  that  any  one  rating  reflects  value  along  several  dimensions  taken 
together.  However,  the  judges’  unweighted  ratings  indicate  where  each  feature 
falls  on  each  separate  dimension  of  value. 
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TABLE  4.  MULTI-ATTRIBUTE  UTILITIES  DATA  FOR  THE  SEVILLE/BURTEK 
DEVICE  AT  BOTH  SCHOOLS 


FEATURE 


TOTAL  VALUE  FEATURE 


TOTAL  VALUE 


Table  5  presents  the  judges*  total  unweighted  ratings  for  features 
in  the  Seville/Burtek  device  along  each  dimension  of  value.  By  using  these 
values,  we  may  determine  how  features  compare  to  each  other  on  each  dimension 
For  purposes  of  this  discussion,  a  value  is  considered  high  or  low  if  it  fall 
at  least  one  standard  deviation,  above  or  below  the  mean. 

For  this  device,  of  all  features  rated,  the  concepts  behind  the  student 
performance  record  and  3D  module  are  viewed  by  our  experts  as  the  most 
important  concepts  in  a  generic  maintenance  training  simulator.  Further,  in 
the  Seville/Burtek  device,  the  implementation  of  concepts  underlying  the  3D 
module  and  student  performance  record  is  good  compared  to  other  features. 

On  the  other  hand,  judges  felt  that  the  concepts  behind  the  student 
response  panel  and  the  implementation  of  concepts  in  that  feature  were  poor 
compared  to  other  features.  In  addition,  judges  felt  that  concept  imple¬ 
mentation  was  poor  for  the  performance  feedback  feature  compared  to  other 
features . 

For  the  operations  dimension,  judges  rated  the  student  performance 
record  as  the  best  and  the  slide  projector  unit  the  worst  feature  as 
compared  to  other  features. 

Finally,  judges  rated  the  3D  module  the  most  motivating  feature  of  the 
Seville/Burtek  device.  The  slide  projector  unit  and  student  response  panel 
were  seen  as  the  least  motivating  features. 

3.  Summary.  The  preceding  two  sections  presented  a  discussion  of 
interviewee  opinions  and  experts'  quantitative  judgments  about  a  selection 
of  features  in  the  Seville/Burtek  device.  In  general  ways,  the  experts' 
judgments  matched  interviewee  opinions  about  device  features. 

Of  all  the  features,  the  3D  module  and  student  performance  record 
received  the  highest  praise  for  design  and  performance.  The  device’s 
troubleshoot  only  mode  also  received  praise  as  a  valuable  and  dependable 
device  feature. 


TABLE  5.  TOTAL  UNWEIGHTED  RATINGS  FOR  FEATURES  IN  THE  SEVILLE/BURTEK 
DEVICE  ALONG  FOUR  DIMENSIONS  OF  VALUE 


FFATURF 

DIMENSION 

CONCEPT 

IMPLEMENTATION 

OPERATION 

MOTIVATION 

3D  module 

510 

530 

465 

470 

Student  performance 

record 

550 

510 

485 

405 

Slide  projector  unit 

355 

440 

275 

280 

Student  response 

panel 

300 

400 

425 

220 

Student  CRT 

355 

430 

455 

320 

Remove/replace 

capability 

425 

460 

380 

445 

Random  malfunction 
selection 

355 

450 

400 

325 

Performance 

feedback 

490 

395 

400 

430 

Troubleshoot  only 
mode 

450 

4 

455 

420 

440 

Mean  421.1 
S.D.  85.1 


452.2 

44.9 


411.7 

61.5 


372.8 

85.9 


Several  features  fell  into  a  middle  range.  Those  features  are  per¬ 
formance  feedback,  remove/replace  capability,  student  CRT,  and  random 
malfunction  selection.  In  the  opinion  of  interviewees  and  experts,  those 
features  were  designed  well  and  performed  adequately. 

Two  features  were  found  to  be  especially  in  need  of  improvement.  Those 
features  were  the  student  response  panel  and  slide  projector  unit.  In 
general,  the  opinion  was  that  the  design  of  the  response  panel  might  be 
improved.  The  main  problems  with  the  slide  projector  unit  were  operational 
ones. 

Interviewees  felt  the  instructor  CRT,  instructor  control  panel  and 
editing  system  were  adequate  in  design  and  performance.  These  features  were 
not  rated  by  the  panel  of  experts. 

C.  Results:  The  Grumman  Device 

1.  Opinions  about  Features.  Information  in  this  section  comes  from 
structured  interviews.  The  interview  protocol  contained  10  device  features 
for  interviewees  to  comment  on.  (It  was  considered  inappropriate  for  device 
contractors  to  comment  on  each  other's  devices.)  In  addition,  the  protocol 
contained  general  questions  about  feature  strengths  and  weaknesses. 

a.  3D  module.  Nearly  all  interviewees  approved  of  the  concepts  behind 
this  feature  and  concept  implementation  as  well.  The  component  parts  in  the 
3D  modules  are  high  fidelity.  Thq  fidelity  of  the  arrangement  of  parts  for 
the  engine  module  is  not  as  high.  However,  this  layout  was  designed  for  easy 
access  to  the  parts,  a  feature  praised  by  several  interviewees.  Interviewees 
felt  the  Grumman  3D  modules  were  "reasonable,"  "adequate,"  and  even  "excellent. 
The  3D  module  also  received  high  marks  for  its  operational  record. 

Several  interviewees  commented  that  this  3D  module  was  highly  interactive 
with  the  rest  of  the  system.  Only  one  felt  that  this  high  degree  of  inter¬ 
action  was  not  required. 


The  3D  module  received  basically  one  criticism.  Some  felt  that  the 
low  fidelity  component  part  arrangement,  with  its  easy  access  design, 
oversimplified  training.  Slower  students  benefited  from  this  arrangement, 
but  faster  students  did  not. 

b.  Student  performance  record.  Almost  all  interviewees  criticized 
this  device  feature  because  too  little  information  was  presented.  Several 
interviewees  added  that  what  was  presented,  was  difficult  to  interpret. 

Grumman  personnel  stated  that  this  feature  was  designed  based  on  the 
recommendations  of  their  instructional  design  staff. 

c.  Instructor  CRT.  Opinions  were  mixed  regarding  the  instructor  CRT. 
Most  interviewees  felt  that  more  information  displayed  on  the  CRT  would  be 
helpful.  Several  interviewees  comnented  that  the  information  available  on 
the  instructor  CRT  concerned  the  system,  not  the  student,  and  that  the 
information  presented  about  the  student's  progress  was  not  especially 
informative.  On  the  other  hand,  several  interviewees  commented  that  this 
feature  worked  well.  One  added  that  the  pre-lesson  system  check  was 
valuable. 

d.  Video  disc.  Most  interviewees  approved  of  the  concept  of  a  video 
disc  feature  in  a  generic  maintenance  training  device.  Benefits  cited 
include  high  resolution  and  longevity  of  discs.  One  interviewee  felt  that 
this  high  technology  feature  made  studying  more  interesting  to  students. 
Several  interviewees  stated  that  concept  implementation  was  good  for  this 
feature. 

Many  interviewees  had  criticisms  of  the  video  disc.  Several  mentioned 
that  the  high  cost  and  difficulty  of  producing  discs  make  the  device  some¬ 
what  inflexible.  New  material  is  not  easily  added.  Some  interviewees  felt 
the  video  disc's  reaction  time  was  too  slow,  and  faster  students  became 
bored.  Some  interviewees  mentioned  the  poor  maintenance  record  of  the 
video  disc.  One  reported  frequent  disc  drive  jams,  overheating  and  problems 


due  to  dust  particles.  Several  interviewees  felt  that  the  video  disc 
feature  is  a  potentially  powerful  feature,  but  in  this  device  it  was  not 
used  to  capacity. 

Grumman  commented  that  the  choice  of  video  disc  was  based  in  part  on 
prior  poor  experiences  with  a  slide  projector  unit  in  a  simulator.  They 
added  that  the  video  disc  concept  is  state-of-the-art  in  video  technology. 

e.  Student  CRT  and  touch  panel .  Text  from  the  video  disc  was  presented 
on  the  student  CRT.  The  touch  panel,  incorporated  into  the  student  CRT,  was 
used  by  the  student  to  enter  responses.  About  half  the  interviewees  felt 
the  touch  panel  was  easy  to  use,  more  user  friendly  than  a  keyboard,  and 
about  half  felt  it  was  unnecessary.  One  commented  that  the  touch  panel  was 
especially  motivating  to  students  because  of  its  novelty.  A  few  commented 
that  students  had  to  touch  the  panel  exactly  in  the  right  spot  for  it  to  work; 
thus,  some  students  sat  very  closely  to  the  CRT  and  too  far  away  from  the  3D 
module. 

Related  to  this  positioning  problem,  one  interviewee  mentioned  that  the 
student  CRT  was  small  and  was  mounted  at  an  angle  such  that  frequent  turning 
was  required  on  the  part  of  the  student  to  use  both  the  CRT  and  the  3D  module. 

f.  Request  help.  This  feature  on  the  student  CRT  allowed  the  student 
to  request  of  the  computer  extra  help  with  the  lesson.  A  few  interviewees 
offered  criticisms:  one  felt  the  review  information  was  not  extensive  enough; 
one  felt  some  students  overused  the  feature;  one  felt  it  worked  too  slowly. 
However,  one  interviewee  felt  the  feature  was  motivating  to  students,  and 
most  interviewees  felt  it  was  a  useful  feature  of  the  device. 

g.  Editing  system.  The  editing  system  allows  an  instructor  to  change 
the  program.  Grumman  commented  that  it  was  designed  for  ease  of  use. 

According  to  interviewees  at  USAADS,  personnel  at  their  school  were  not 
taught  how  to  use  the  Grumman  editing  system.  Only  very  limited  training 
occurred  at  USAOC&S.  Thus,  the  editing  system  was  rarely  used  during  the 
evaluation. 


h.  Repeat  lesson  option.  A  student  could  repeat  an  entire  lesson 

by  using  this  feature.  Grumman  stated  that  a  positive  aspect  of  this  feature 
is  that  it  decreases  dependence  on  the  instructor. 

Most  interviewees  felt  that  the  concept  of  this  feature  is  good  and 
necessary  in  self-paced  instruction.  Two  interviewees  observed  that  few 
students  used  the  option.  One  interviewee  consented  that  simply  repeating 
a  lesson  may  not  be  adequate  remedial  instruction.  However,  the  majority 
felt  the  feature  worked  well. 

i.  Call  instructor  after  two  errors.  This  feature  automatically 
stopped  the  device  when  the' student  made  two  sequential  errors.  Grumman 
commented  that  this  feature  was  recommended  by  its  instructional  system 
developers.  Many  interviewees  liked  the  concept  behind  the  feature.  One 
interviewee  commented  that  this  feature  helped  students  proceed  in  a  produc¬ 
tive  manner. 

The  main  criticism  of  this  feature  was  that  it  sometimes  scored  errors 
incorrectly.  Thus,  it  would  occasionally,  in  error,  lock  students  out  of 
the  program.  Further,  students  did  not  know  how  to  get  back  into  the  program. 
Small  errors  could,  perhaps  unnecessarily,  activate  the  feature.  So,  in  many 
cases,  the  feature  fostered  unnecessary  dependence  on  the  instructors. 

j.  Performance  feedback.  Performance  feedback  was  given  to  the  student 
on  the  student  CRT.  Most  interviewees  felt  that  the  concept  was  good.  Many 
gf  the  interviewees  commenting  found  the  buzzer,  which  was  a  signal  for  the 
student  to  look  at  the  CRT  for  feedback,  distracting.  A  few  interviewees 
said  the  buzzer  mechanism  occasionally  malfunctioned.  Some  students 
incorrectly  thought  the  buzzer  was  a  punisher. 

k.  Software.  Grumman  device  programming  is  written  in  Assembler 
language.  Grumman  chose  this  language  based  on  previous  good  experience 
with  it. 

Two  problems  with  inflexibility  of  software  were  mentioned  several 
times.  One  problem  related  to  the  fact  that  an  instructor  could  only 
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begin  a  lesson  at  its  beginning.  Another  related  to  the  fact  that  it 
was  difficult  for  an  instructor  to  insert  a  malfunction. 

2.  Expert  Ratings.  A  panel  of  four  experts  was  asked  to  rate  features 
of  the  Grumman  device.  The  ratings  were  then  subjected  to  a  multi -attribute 
utilities  analysis.  Basically,  each  design  feature  was  rated  along  each  of 
four  dimensions  of  value:  concept,  implementation,  operation,  and  motivation 
Procedures  used  for  rating  and  data  analysis  are  described  in  section  II. A. 2. 
above. 

Table  6  presents  each  judge's  weighted  rating  and  the  total  rating  for 
each  device  feature.  The  data  in  Table  6  are  grand  totals  of  the  weighted 
ratings  for  this  device  at  both  schools.  Thus,  the  data  in  Table  6  suggest 
how  valued  each  device  feature  is,  taking  all  four  dimensions  into  account. 
Features  in  Table  6  are  listed  in  descending  order  by  rank.  As  mentioned 
earlier,  the  third  judge's  ratings  are  a  combination  of  the  ratings  from  the 
expert  who  rated  the  devices  at  USAADS  and  the  expert  who  rated  the  devices 
at  USAOC&S . 

Table  6  shows  that  the  most  valued  feature  of  the  Grumman  device  was 
the  3D  module;  its  value  exceeded  one  standard  deviation  above  the  mean  of 
this  distribution.  The  performance  feedback  feature  also  earned  a  high 
value;  its  value  was  very  nearly  one  standard  deviation  above  the  mean. 

Two  features  scored  values  less  than  one  standard  deviation  below  the 
mean:  the  student  performance  record  and  the  mandatory  instructor  call 
after  two  errors. 

Table  7  presents,  for  both  schools,  aggregate  ratings  for  each  feature. 
The  judges'  ratings  indicate  that  features  of  the  Grutmian  device  at  USAOC&S 
were  similar  in  value  to  features  on  the  device  at  USAADS;  however,  one 
important  difference  emerged.  As  shown  in  Table  7,  the  most  valued  feature 
at  USAADS  was  the  3D  module,  yet  this  feature  was  only  third  in  value  at 
USAOC&S.  The  request  help  feature  was  the  most  valued  feature  on  the  device 
at  USAOC&S.  For  both  devices,  performance  feedback  was  the  second  most 
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TABLE  6.  MULTI-ATTRIBUTE  UTILITIES  DATA  FOR  THE  GRUMMAN  DEVICE 


TABLE  7.  MULTI -ATTRIBUTE  UTILITIES  DATA  FOR  THE  GRUMMAN  DEVICE 
AT  BOTH  SCHOOLS 


USAOC&S 

USAADS 

FEATURE 

TOTAL  VALUE 

FEATURE 

TOTAL  VALUE 

Request  help 

17,731 

3D  module 

21,294 

Performance  feedback 

17,465 

Performance  feedback 

19,568 

3D  module 

16,000 

Request  help 

18,407 

Touch  panel 

15,471 

Video  disc 

17,559 

Repeat  lesson  option 

14,464 

Repeat  lesson  option 

15,104 

Video  disc 

12,684 

Touch  panel 

15,015 

Student  performance 
record 

11,452 

Student  performance 
record 

11,952 

Call  instructor 
after  two  errors 

10,960 

Call  instructor 
after  two  errors 

7,389 

Mean  =  14,528.4 
S.D.  =  2,606.0 


Mean  =  15,768.0 
S.D.  *  4,486.9 


valued  feature;  its  rating  at  USAOC&S  was  more  than  one  standard  deviation 
above  the  mean  and  the  value  for  the  feature  at  USAADS  was  nearly  one 
standard  deviation  above  the  mean. 

For  both  Grumman  devices,  the  student  performance  record  and  the 
mandatory  call  instructor  after  two  errors  feature  were  the  two  least 
valued  features.  The  values  for  both  features  on  the  USAOC&S  devices 
fell  more  than  one  standard  deviation  below  the  mean.  On  the  USAADS 
device,  the  instructor  call  feature  value  fell  below  one  standard  devia¬ 
tion  below  the  mean. 

Two  features,  the  instructor  CRT  and  the  editing  system,  were  included 
in  the  ratings  protocol.  However,  these  features  were  not  included  in  the 
analysis  because  judges  felt  the  dimension  of  "motivation"  did  not  apply  to 
these  features.  In  addition,  personnel  were  not  systematically  trained  to 
use  the  editing  system,  so  ratings  on  the  dimensions  of  implementation  and 
operations  were  omitted  by  some  judges.  Thus,  data  for  these  two  features 
are  incomplete. 

As  mentioned  earlier,  ratings  in  a  multi -attribute  utilities  analysis 
are  based  on  all  dimensions  of  value  taken  together.  However,  the  judges' 
unweighted  ratings  indicate  where  each  feature  falls  on  each  separate 
dimension  of  value. 

Table  8  presents  the  judges'  total  unweighted  ratings  for  features 
in  the  Grumman  device  along  each  dimension  of  value.  Using  these  values, 
we  may  determine  how  features  compare  to  each  other  on  each  dimension.  A 
value  is  considered  high  or  low  if  it  falls  at  least  one  standard  deviation 
above  or  below  the  mean  of  the  distribution. 

As  shown  in  Table  8,  judges  considered  the  concepts  underlying  the  3D 
module  more  important  than  concepts  behind  other  features.  Judges  also 
felt  that  concept  implementation  was  better  for  the  3D  module  than  for  other 
features . 
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FEATURE 


DIMENSION 


CONCEPT  IMPLEMENTATION  OPERATION  MOTIVATION 


3D  module 

480 

370 

260 

360 

Student  performance 
record 

400 

-  90 

240 

90 

Video  disc 

370 

275 

210 

375 

Touch  panel 

290 

325 

270 

370 

Request  help 

340 

350 

410 

330 

Repeat  lesson  option 

320 

280 

330 

200 

Mandatory  instructor 
call  after  two  errors 

220 

195 

160 

130 

Performance  feedback 

430 

300 

360 

370 

Mean 

356.3 

273.1 

267.5 

278.1 

S.D. 

82.3 

91.3 

92.5 

119.0 
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Judges  felt  that  the  concepts  behind  the  mandatory  instructor  call 
feature  were  less  valuable  than  for  other  features.  Judges  also  felt 
that  this  feature  was  low  on  both  the  operations  and  motivation  dimensions. 

Judges  rated  the  request  help  feature  higher  than  other  features  on 
the  operations  dimension. 

The  student  performance  record  was  rated  lower  than  other  features 
on  the  motivation  dimension. 

3.  Summary.  The  preceding  two  sections  presented  a  discussion  of 
interviewee  opinions  and  experts'  quantitative  judgments  about  a  selection 
of  features  in  the  Grumman  device.  Based  on  the  data  presented,  general 
suimary  statements  about  the  features  may  be  made. 

Of  all  the  features  in  the  device,  the  3D  module  received  the  most 
praise  for  design  and  performance. 

Five  features  seemed  to  fall  in  a  middle  range.  The  repeat  lesson 
option,  request  help,  touch  panel  and  performance  feedback  features  were 
viewed  as  adequate  in  design  and  performance.  The  video  disc  was  viewed 
as  excellent  conceptually,  but  had  some  problems  practically. 

Two  features  seemed  in  special  need  of  improvement:  the  mandatory 
instructor  call  after  two  errors  and  the  student  performance  record. 

The  former  was  felt  to  foster  dependence  on  the  instructor  and  perhaps 
decrease  motivation.  Further,  operational  problems  are  associated  with 
the  feature.  The  student  performance  record  was  seen  as  inadequate  because 
so  few  student  progress  data  were  presented. 

Interviewees  felt  that  the  instructor  CRT  was  adequate,  but  might  be 
improved  in  comprehensiveness.  Personnel  did  not  test  the  editing  system 
because  they  were  not  trained  to  do  so. 
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D.  Summary 


Sections  II. 8.  and  II. C.  above  have  presented  a  discussion  of  individual 
device  features.  In  this  section,  comments  are  presented  concerning  the  overall 
functioning  and  uses  of  the  devices. 

Interviewees  were  asked  how  well  the  components  of  each  device  were 
integrated  into  a  working  training  system.  Agreement  was  widespread  that 
both  devices  function  well  as  systems. 

Three  points  relevant  to  device  component  integration  were  made.  First, 
one  interviewee  mentioned  that  the  AMTESS  concept  of  integrating  2D  and  3D  com¬ 
ponents  by  computer  is  expensive.  Further,  for  many  basic  MOSs,  that  level 
of  sophistication  may  be  unnecessary.  In  addition,  during  some  lessons,  a 
student  may  spend  most  of  his  or  her  time  watching  the  CRT  or  reading  techni¬ 
cal  manuals;  during  this  time  the  costly  integration  of  2D  and  3D  components 
is  unused.  Thus,  it  may  be  that  this  complex  a  device  is  unnecessary  on  a 
widespread  basis.  Research  on  this  problem  might  test  the  efficiency  and 
effectiveness  of  a  networked,  real  time  system  that  would  allow  one  student 
to  work  on  the  2D  component  while  another  student  uses  the  3D  component. 

Related  to  this  point,  another  interviewee  commented  that  the  Grumman 
components  were  so  tightly  Integrated,  that  if  the  video  disc  was  down, 
students  could  not  progress.  However,  if  Seville/Burtek’s  slide  projector 
was  down,  students  could  progress  using  the  technical  manuals  for  text 
presentation.  This  interviewee  felt  that  integration  might  be  regarded 
an  undesirable  quality  given  the  high  frequency  of  disrepair  for  both  devices. 

Third,  one  interviewee  noted  that  the  Grumman  device  contains  one 
computer  for  the  2D  component  and  a  second  computer  with  each  3D  module. 

In  contrast,  the  Seville/Burtek  device  contains  one  computer  which  operates 
the  2D  component  and  any  3D  module.  Further  study  is  in  order  concerning 
the  economy  and  efficiency  of  one  versus  two  computers  per  device. 


Next,  several  school  personnel  and  the  SAI  data  collectors  gave  their 
opinions  on  how  best  each  simulator  could  be  used  in  order  to  gain  maximum 
benefits  from  it.  The  majority  of  interviewees  felt  that  both  devices 
would  function  best  as  supplemental  or  remedial  training.  Only  one  inter¬ 
viewee  felt  that  the  Seville/Burtek  device  might  directly  replace  actual 
equipment  training. 

Interviewees  gave  several  reasons  for  their  opinions  about  optimal 
use  of  the  Seville/Burtek  device.  Two  interviewees  commented  that  the 
troubleshooting  exercises  are  geared  toward  entry  level  students.  Indeed, 
as  presented  earlier  in  Table  1,  Burtek  personnel  designed  the  device  to 
accommodate  "low-end”  skills  training.  One  interviewee  comnented  that 
some  advanced  students  found  some  remove/ replace  exercises  elementary 
and  of  too  low  fidelity  to  be  useful.  One  interviewee  added  that  some 
critical  tasks  are  not  taught  by  the  devices.  Thus,  for  several  reasons, 
interviewees  felt  that  the  Seville/Burtek  device  would  best  be  used  as 
supplemental  or  remedial  training  for  slow  students. 

As  for  the  Grumman  device,  several  interviewees  commented  that  this 
device  was  geared  toward  remedial  training  because  of  the  slow  rate  of 
material  presentation.  Others  noted  that  because  of  limited  remove/ replace 
capability,  this  device  functions  as  supplemental  training.  As  with 
Seville/Burtek 's  device,  some  critical  tasks  are  not  taught  with  the 
Grumman  device. 

Most  interviewees  felt  that  the  two  contractors  had  indeed  designed 
and  constructed  devices  whose  components  were  integrated,  functioning, 
and  at  some  level,  effective  in  training.  Certainly,  both  contractors 
were  pleased  with  their  devices.  Perhaps  the  greatest  problem  with  both 
devices  is  their  dependability.  Down-time  plagued  the  device  evaluation. 
Certainly,  the  maintenance  records  of  these  breadboard  devices  would  be 
unacceptable  in  a  production  model.  On  the  other  hand,  it  may  have  been 
somewhat  unrealistic  to  subject  breadboard  models  to  transfer  of  training 
studies.  In  general,  however,  participants  felt  that  these  devices  were 
adequate. 
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III.  DISCUSSION 

This  report  has  included  two  discussions.  First,  the  development  of 
AMTESS  devices  was  presented  and  second,  a  summary  of  opinions  about  the 
features  of  two  AMTESS  devices  was  presented. 

A.  Device  Development 

AMTESS  device  development  grew  out  of  the  Army's  need  for  simulators 
to  replace  or  supplement  actual  equipment  used  in  maintenance  training. 

AMTESS  device  conceptual  designs  were  first  developed,  then  two  breadboard 
devices  were  constructed.  Problems  arose  during  device  construction 
stemming  from  changes  in  government  requirements  for  the  devices,  partic¬ 
ularly  in  the  school  settings.  Due  to  serious  lack  of  communication 
between  the  government  and  the  contractors,  the  devices  did  not  meet  all 
the  schools'  requirements.  The  delivered  devices  were  subjected  to  formal 
experimental  evaluation,  but  the  evaluation  did  not  proceed  as  planned. 

The  evaluation  suffered  because  the  devices  did  not  perform  as  the  schools 
and  the  evaluation  team  had  expected  and  because  adequate  numbers  of  subjects 
were  not  available  for  testing. 


The  problem  encountered  in  device  development  reflected  unclear  communi¬ 
cations  among  the  agencies  involved.  Some  agencies  communicated  frequently 
among  themselves  and  other  communicated  less  often,  yet  regardless  of  the 
communication  frequency,  costly  misunderstandings  occurred.  Communication 
between  project  administration  and  those  actually  developing  and  testing 
devices  seemed  especially  poor.  Although  many  interview  participants  noted 
that  administrators  were  always  available  by  phone,  the  contacts  with  adminis¬ 
tration  in  many  cases  did  not  produce  concrete  plans. 

One  area  in  which  high-level  adminstrative  support  seemed  lacking  was 
with  the  schools.  Fortunately,  personnel  at  each  test  site  school  were 
involved  in  AMTESS  for  several  years.  Unfortunately,  however,  high-level 
school  administrative  personnel  were  not  involved.  In  addition,  many 
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different  types  of  school  personnel  were  involved  (instructors,  course  writers, 
subject  matter  experts,  for  example),  but  they  were  not  all  under  the  authority 
of  the  schools'  points-of-contact  for  AMTESS.  When  changes  occurred  in  the 
schools*  curricula  or  training  procedures,  the  AMTESS  representative  may  not 
have  been  in  a  position  to  counter  the  changes. 

Not  only  was  administrative  support  concerning  agency  assignments  and 
coordination  frequently  lacking,  but  financial  support  may  have  been  a 
problem  as  well.  As  government  device  requirements  changed,  so  contractor 
financial  requirements  increased.  In  addition,  limited  travel  funds 
restricted  some  agencies  from  monitoring  device  development  and  evaluation 
as  needed. 

Kane  and  Holman  (1982)  applied  the  Army  Life  Cycle  System  Management 
Model  to  the  process  of  developing  and  acquiring  training  devices.  Kane 
and  Holman  (1982)  identified  seven  important  steps  in  the  training  device 
process: 

1.  Gather  data  on  existing  training  systems;  select  tasks 
to  be  taught;  select  training  modes. 

2.  Award  contract  for  device  concept  development. 

3.  Demonstrate  and  validate  device-based  training  concepts. 

4.  Improve  device  requirements  based  on  #3. 

5.  Develop  detailed  predictions  of  the  training  effective¬ 
ness  of  alternative  device  models;  award  contract  to 
build  the  device. 

6.  Major  test  of  prototype  simulator  transfer-of-training 
effectiveness. 

7.  Construct  and  evaluate  production  model  device. 

In  the  case  of  the  AMTESS  procurement.  Step  1  was  accomplished  in  1978 
by  the  Seville  maintenance  training  survey.  PM  TRADE  completed  Step  2  in 
1979.  In  1980,  four  contractors  submitted  final  design  reports  based  on 
front  end  analyses.  As  for  Step  3,  the  four  contractors'  reports  demonstrated 
concepts,  but  did  not  empirically  validate  them.  The  extent  to  which  the 
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government  completed  Step  4  (improving  device  requirements)  is  unknown, 
but  the  Phase  II  device  construction  proposals  were  submitted  before 
final  Phase  I  reports  were  submitted.  In  light  of  the  problems  that 
developed  surrounding  task  selection,  too  little  attention  to  Step  4 
activities  may  have  prevented  the  evaluation  from  proceeding  as  originally 
designed. 

In  the  AMTESS  project,  some  Step  5  prediction  was  accomplished. 

A  new  type  of  prediction  method  was  tested  although  this  did  not  occur 
until  after  device  construction  (Klein,  1982).  The  government  awarded 
device  construction  contracts  in  September  1980. 

The  Step  6  device  evaluation  was  completed  in  1983.  In  the  case  of 
the  AMTESS  devices,  further  experimentation  is  planned.  Production  models 
have  not  been  ordered. 

B.  Device  Features 

The  problems  in  communication  during  the  AMTESS  device  development  are 
not  clearly  related  to  opinions  about  the  features  of  the  devices.  However, 
two  substantial  problems  with  the  devices  noted  by  participants  may  relate 
to  unmatched  expectations  between  the  contractors  and  the  government. 

One  of  these  problems  concerns  the  Grumman  device's  student  performance 
record.  Nearly  all  participants  felt  that  this  record  is  inadequate  because 
it  gives  too  little  information.  Perhaps  this  performance  record  would  have 
been  more  complete  had  Grumman  been  more  aware  of  government  requirements  in 
this  regard. 

The  second  and  much  more  important  problem  concerns  the  high  frequency 
with  which  the  devices  were  in  disrepair  during  device  evaluation.  The 
contractors  were  supposed  to  construct  first-run,  breadboard  models,  but 
at  the  same  time,  the  devices  should  have  been  strong  enough  to  withstand 
testing.  Perhaps  rigorous  testing  should  not  be  planned  for  breadboard 
devices,  or  provisions  for  ensuring  device  reliability  and  maintainability 
should  be  implemented. 


Comments  made  by  some  interview  participants  suggested  factors  that 
might  be  considered  in  designing  device  features.  First,  if  devices  are 
designed  for  entry  level  students  for  low-skill  MOSs,  high  physical 
fidelity  of  parts  and  ease  of  access  are  desirable.  As  a  Burtek  repre¬ 
sentative  pointed  out,  advanced  students  do  not  need  to  practice  simple 
tasks  on  an  expensive  simulator;  those  students  often  profit  from  reading 
or  following  schematics.  Devices  designed  for  entry  level,  low-skill  MOSs 
should  provide  ample  opportunity  for  direct  shaping  of  the  skill  by  the  device. 
In  addition,  instructional  material  for  low-skill  MOSs  may  need  to  be  in 
simpler  English  than  material  for  more  advanced  MOSs. 

The  level  of  student  and  MOS  level,  considerations  mentioned  above  raise 
the  issue  of  how  generic  a  maintenance  training  simulator  can  be.  Some 
interview  participants  suggested  that  the  simulator  be  generic  across  skills, 
not  across  specific  MOSs.  Simulator  training  designed  in  this  way  would  be  ’ 
supplemental  to  conventional  Army  courseware  which  is  designed  around  MOSs. 

It  may  be  necessary  in  a  generic  skill  simulator  to  reduce  physical  fidelity. 

The  usefulness  of  a  low  fidelity  simulator  should  be  carefully  examined  with 
respect  to  entry  level,  low-skill  MOS  students.  Generally,  training  for  these 
students  needs  to  be  concrete. 

The  question  concerning  the  relative  importance  of  troubleshoot  (diagnosis) 
versus  remove/replace  (repair)  can  only  be  answered  by  examining  the  skills  or 
MOSs  to  be  taught  by  the  device.  The  front  end  analysis  would  determine  which 
of  these  features  should  be  included  in  the  device. 

Because  the  devices  could  not  be  fairly  compared  to  each  other,  the  merits 
of  individual  features  were  not  assessed.  Research  on  the  efficacy  of  device 
features  is  planned. 


IV.  CONCLUSIONS 


This  section  presents  guidelines  for  future  device  development.  These 
guidelines  are  based  on  the  AMTESS  experiences  discussed  in  this  report. 

1.  Project  leadership  should  take  an  active  role  in  overseeing 
the  activities  of  all  agencies  involved.  Meetings  should  have  specific 
agendas.  Minutes  of  all  meetings  should  be  taken  and  distributed  to  all 
participants.  Minutes  should  be  thorough  and  spell  out  who  should  do  what 
next. 

Documentation  of  project  activities  is  particularly  important  when 
projects  are  lengthy,  personnel  change  frequently,  and  participants  are 
separated  by  long  geographical  distances.  Such  was  the  case  with  the 
AMTESS  project.  Several  participants  in  the  AMTESS  project  mentioned  that 
the  JWG  would  have  been  more  successful  in  its  coordination  of  AMTESS 
activities  had  specific  minutes  been  taken  at  JWG  meetings.  Another 
instance  where  documentation  was  apparently  lacking  in  the  AMTESS  project 
were  the  contacts  between  school  and  contractor  personnel.  Such  documenta¬ 
tion  if  made  available  to  the  JWG  might  have  alerted  the  JWG  to  the  problems 
discovered  at  device  delivery. 

2.  Formal  memorandums  of  understanding  should  be  signed  which  officially 
link  all  participants  needed  in  the  project.  The  responsibilities  of  each 
organization,  both  government  and  contractor,  should  be  clear  to  all  participants. 

In  the  AMTESS  project,  cooperative  agreements  were  needed  in  at  least 
three  directions.  First,  participants  in  AMTESS  Phases  I  and  II,  notably  the 
schools,  contractors,  and  PM  TRADE,  needed  some  agreement  to  guide  hardware 
and  software  development.  Second,  participants  in  that  portion  of  SIMTRAIN  I 
which  included  device  evaluation  needed  an  agreement  concerning  how  they  would 
conduct  the  evaluation.  Those  participants  were  ARI,  schools,  and  the  evalua¬ 
tion  team.  Finally,  the  evaluation  effort  needed  an  official  link  to  AMTESS 
Phase  II.  In  the  AMTESS  project,  this  third  link  may  have  been  the  weakest 
connection.  As  discussed  earlier  in  this  report,  evaluation  plans  were 
developed  without  contractor  input. 
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3.  The  coordinating  group  should  include  personnel  with  authority 
over  people  whose  cooperation  is  needed  in  each  participating  agency. 

As  mentioned  by  one  interviewee,  problems  in  this  regard  occurred  in 
the  AMTESS  project  with  the  schools.  In  many  instances,  important  work 
was  performed  without  proper  supervision. 

4.  Project  administration  should  plan  with  flexibility  to  accommodate 
disruptions . 

Problems  in  the  AMTESS  project  resulted  in  device  delivery  delays 
which  affected  the  evaluation  time  schedule  and  created  a  need  for  additional 
funds.  When  changes  were  required  in  the  tasks  to  be  taught  by  the  devices, 
no  administrative  mechanism  was  in  place  to  direct  the  changes.  As  discussed 
earlier,  the  contractors  and  schools  planned  major  device  changes  and  informed 
the  other  organizations  later.  A  preplanned  administrative  mechanism  for  making 
unexpected  changes  might  have  produced  decisions,  with  better  consequences 
than  was  the  case  with  AMTESS. 

Several  interviewees  commented  that  disruptions  created’ the  need  for 
additional  money.  •  Provisions  should  be  made  for  modification  in  contracting 
procedures  to  facilitate  compatibility  with  changing  device  requirements. 

5.  Frequent  and  precise  communication  should  characterize  the  relation¬ 
ship  between  the  government,  contractors,  and  the  evaluation  team. 


Many  interviewees  felt  that  communication  was  a  problem  in  AMTESS.  As 
several  interviewees  noted,  for  example,  the  device  contractors  knew  little 
about  the  evaluation  process,  and  the  evaluation  team  knew  little  about  the 
devices.  Perhaps  an  expedient  course  would  have  been  to  require  frequent 
communication  between  the  contractors  and  the  evaluation  team  to  ensure  that 
evaluation  plans  were  appropriate  and  fully  understood. 
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6.  The  government  should  carefully  choose  the  MOSs  from  which  tasks 
are  selected  for  inclusion  in  the  devices.  Those  MOSs  should  be  stable. 
Preferably,  complete  performance  tests  would  be  available  for  those  MOSs 
and  be  in  use  in  conventional  training. 

The  Phase  II  disruptions  described  earlier  illustrate  the  point  that 
selection  of  MOSs  and  tasks  to  be  taught  turned  out  to  be  one  of  the  largest 
problems  in  the  AMTESS  project.  Had  the  MOSs  not  changed  during  the  project, 
much  of  the  confusion  surrounding  task  selection  might  have  been  avoided. 

Complete  performance  tests  did  not  exist  for  the  tasks  taught  by  the 
devices.  The  available  performance  tests,  however,  were  helpful  in 
designing  the  comprehensive  tests  necessary  to  study  transfer  of  training 
in  the  device  evaluation. 

7.  Precise  device  specifications  should  be  made  available  to  the  device 
contractors.  These  specifications  should  include  a  general  list  of  device 
features,  the  intended  use  of  the  simulator  in  training,  the  level  of  students 
involved,  and  the  demands  of  any  planned  evaluations. 

As  representatives  of  all  involved  organizations  noted,  the  AMTESS 
project  was  a  research  and  development  effort.  For  Phase  I,  many  device 
requirements  were  not  specified.  Specifications  were  needed,  however, 
for  Phase  II  devices.  The  contractors  needed  more  information  about  how 
devices  should  perform  to  ensure  a  successful  evaluation. 

8.  An  explicit  mechanism  for  quality  control  of  the  devices  should  be 
instituted.  This  mechanism  should  track  devices  from  construction,  to 
delivery,  through  evaluation. 

Device  construction  should  be  carefully  monitored  both  by  sponsoring 
agency  and  user  agency  personnel.  Careful  monitoring  should  ensure  that 
the  devices  meet  all  requirements . 
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Careful  AMTESS  device  construction  monitoring  might  have  been  expen¬ 
sive  because  of  the  geographical  locations  of  the  organizations  involved. 

As  device  requirements  changed,  monitoring  should  have  increased.  Instead, 
as  mentioned  earlier,  no  one  had  responsibility  for  carefully  checking  that 
the  devices  were  being  appropriately  changed. 

The  quality  control  mechanism  should  ensure  that  devices  are  designed 
for  dependability  and  ease  of  maintainability.  Devices  should  be  sturdy 
enough  to  withstand  testing.  As  discussed  above,  many  device  features  were 
criticized  for  poor  maintenance  records.  Disrepair  caused  delays  in  the 
device  evaluation.  (More  information  regarding  device  malfunctioning  will 
be  presented  in  the  SIMTRAIN  I  Extension  final  report.) 

9.  The  criteria  for  device  acceptance  should  be  comprehensive,  precise, 
and  fully  understood  by  all  parties.  If  criteria  are  not  met,  the  devices 
should  not  be  accepted. 

In  the  AMTESS  project,  the  weaknesses  in  acceptance  criteria  were  in 
the  area  of  task  selection  and  device  dependability.  The  absence  of  acceptance 
criteria  in  the  areas  of  maintainability  and  reliability  effectively  forces  the 
government  to  accept  devices  regardless  of  their  performance  along  these 
dimensions. 

10.  Instructor  training  in  the  use  of  the  devices  should  be  required 
and  monitored. 

As  mentioned  earlier,  instructor  training  was  not  provided  for  the 
Grumman  device  at  USAADS,  and  only  brief  training  for  personnel  at  USAOC&S. 

One  unfortunate  consequence  of  this  omission  was  that  instructors  did  not 
use  the  Grumman  editing  system  and  it  could  not  be  evaluated. 

Although  numerous  problems  occurred,  the  AMTESS  project  may  be  judged 
successful  in  at  least  three  very  important  ways.  First,  functioning, 
effective  devices  were  constructed.  These  devices  did  in  fact  provide 


substantial  capability  for  maintenance  training  beyond  existing  approaches 
Second,  the  concept  of  generic  maintenance  training  device  development  and 
implementation  appears  vindicated.  Third,  and  perhaps  as  important,  the 
project  offers  valuable  lessons  in  how  to  improve  similar  future  device 
developmental  efforts. 
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HISTORY 


Target 

1.  When  did  you  begin  work  on  AMTESS?  ALL 

-  What  was  your  role? 

-  Did  it  change?  How? 

2.  Have  you  participated  in  any  activities  that  led  to  the  ALL  but  SAI 

development  of  the  AMTESS  concept? 

-  If  yes,  what  were  these  activities? 

3a.  What  do  you  believe  are  the  overall  objectives  of  the  ALL 

AMTESS  initiative? 

-  To  what  extent  are  these  objectives  understood 
by  other  organizations  involved  in  AMTESS? 

-  What  changes,  if  any,  have  there  been  in  these 
objectives  over  time?  Why? 

-  What  impacts  resulted? 

3b.  What  do  you  believe  are  the  objectives  of  the  AMTESS  ALL 

procurement? 

-  To  what  extent  do  other  organizations  involved 
in  AMTESS  share  these  objectives? 

-  Have  these  objectives  changed  for  you  or  any 
other  organization  during  the  course  of  AMTESS? 

-  If  so,  how  and  why? 

4a.  Please  identify  what  you  consider  to  be  critical  tasks  ALL 

In  the  Implementation  of  AMTESS.  Identify  both  critical 
tasks  which  your  organization  performed  and  which  other 
organizations  Involved  in  AMTESS  performed. 

(Probe:  Ask  respondents  to  evalute  performance  of 
each  task  named.) 

4b.  List  of  critical  tasks.  Have  respondents  evaluate  ALL 

performance  of  each  task  not  discussed  in  4a. 

A.  PM  TRADE 

1.  Develop  AMTESS  concept. 

2.  Write  SOWs  for  AMTESS. 

3.  Review  proposals. 

4.  Monitor  development  of  devices  at  Tulsa/Bethpage. 

5.  Review  contractor  reports. 

6.  Monitor  device  evaluation. 
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B.  SCHOOLS 


Target 

ALL 


1.  Review  proposals. 

2.  Work  with  contractors  during  FEAs. 

3.  Monitor  development  of  devices  at  Tulsa/Bethpage. 

4.  Government  acceptance  of  devices  at  Ft.  Bliss/Aberdeen 
Proving  Grounds. 

5.  Assist  ARI/SAI  in  evaluating  devices. 

C.  AR1 

1.  Review  proposals  (contractors  and  Honeywell). 

2.  Prepare  SOW  for  device  evaluation. 

3.  Develop  test  plan. 

4.  Monitor  development  of  devices  at  Tulsa/Bethpage. 

5.  Monitor  device  evaluation. 

D.  ATSC 

1.  Review  proposals. 

2.  Monitor  development  of  devices  at  Tulsa/Behtpage. 

3.  Monitor  device  evaluation. 

E.  CONTRACTORS 

1 .  Perform  FEAs . 

2.  Develop  PSED. 

3.  Develop  life  cycle  cost  estimate. 

4.  Develop  hardware. 

5.  Develop  software. 

6.  Maintain  devices  during  evaluation. 

F.  SAI 

1.  Develop  performance  tests. 

2.  Modify  test  plan. 

3.  Collect  qualitative  data. 

4.  Collect  quantitative  data. 

5.  Were  there  any  tasks  you  consider  critical  to  AMTESS  which  ALL 

were  not  performed  at  all?  4 

(Probe:  Why  were  they  not  performed?) 

6.  In  your  opinion,  which  organization  was  actually  responsible  ALL 
for  managing  the  AMTESS  effort? 

-  Did  they  manage  it  well? 

-  Did  they  have  adequate  authority  and  resources  to 
manage  the  project  well? 
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Target 

7a.  How  would  you  characterize  the  coordination  among  the  ALL 

various  organizations  involved  in  AMTESS? 

7b.  Please  describe  the  nature  of  your  coordination  with  each  ALL 
of  the  organizations  you  were  involved  with  in  AMTESS  and 
evaluate  the  effectiveness  of  that  coordination. 

1 .  ARI 

2.  Schools 

3.  PM  TRADE 

4.  ATSC 

5.  SAI 

6.  contractors 


7c.  Please  describe  the  effectiveness  of  the  Joint  Working  ALL 

Group  meetings  for  planning  and  coordinating  AMTESS 
activities. 

8a.  Please  describe  how  the  tasks  to  be  used  during  the  ALL 

device  evaluation  were  selected. 

8b.  Who  had  the  principal  responsibility  for  selecting  ALL 

these  tasks? 

8c.  What  coordination  was  involved  in  this  task  selection  ALL 

procedure? 

-  What  made  that  coordination  effective/ineffective? 

9.  What  are  your  opinions  about  the  evaluation  of  the  AMTESS  ALL 


devices  being  conducted  at  Aberdeen  Proving  Grounds  and 
Ft.  Bliss? 

o  Valid  evaluation  of  transfer  of  training? 
o  Comprehensive? 

o  Suggestions  for  improving  this  type  of  evaluation? 

10.  I  would  like  you  to  comment  on  the  performance  of  each  of  ALL 
the  organizations  involved  in  AMTESS.  Please  describe 

what  they  did  well  and  what  they  did  poorly. 

1 .  ARI 

2.  Schools 

3.  PM  TRADE 

4.  ATSC 

5.  SAI 

6.  Contractors 

11.  Do  you  feel  that  the  AMTESS  concept,  embodied  in  the  ALL 

prototypes  has  lived  up  to  expectations? 
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2.  Is  the  AMTESS  concept  worth  pursuing?  ALL 

3.  How  does  the  AMTESS  procurement  process  compare  to  that  ALL 

of  other  training  devices? 


FRONT  END  ANALYSIS 

1.  What  do  you  believe  were  the  objectives  of  the  front  end  ALL 

analysis? 

-  To  what  extent  do  other  organizations  share 
these  objectives? 

-  Did  these  objectives  change  for  you  or  other 
organizations  during  the  project? 

2.  Describe  your  organization's  involvement  in  the  front  end  ALL 
analysis  activities. 

3a.  Describe  the  activities  conducted  during  each  of  the  ALL 

following  phases  of  the  FEA. 

-  Task  analysis 

-  Training  requirement  analysis 

-  Fidelity  analysis 

3b.  Did  your  company  perform  any  FEAs  that  were  not  called  CONTRACTORS 

for  in  the  SOW? 

* 

3c.  Were  the  same  personnel  involved  in  all  of  the  FEAs?  CONTRACTORS 

3d.  What  are  the  backgrounds  of  the  individuals  who  conducted  CONTRACTORS 

the  analysis? 

3e.  How  were  the  results  of  the  various  analyses  integrated  CONTRACTORS 

into  the  design  of  the  simulator? 

3f.  How  important  was  the  FEA  in  the  selection  of  specific  CONTRACTORS 

device  features? 

3g.  What  procedures/decisions/mechanisms  insured  that  CONTRACTORS 

requirements  identified  in  the  FEA  would  be  addressed 
by  the  simulator? 

3h.  How  closely  did  the  results  of  the  FEA  match  up  with  CONTRACTORS 

the  final  system  design? 

4.  How  would  you  evaluate  the  performance  of  your  organization  CONTRACTORS 
and  the  performance  of  other  organizations  in  each  of  the 
tasks  of  the  FEA? 


Target 

CONTRACTORS 


5.  Were  there  any  aspects  of  the  FEA  which  you  consider 
critical  which  were  not  performed  at  all? 

6.  Describe  the  extent  to  which  you  and  your  organization  ALL 

were  able  to  coordinate  FEA  activities  with  other  organiza¬ 
tions. 

7.  How  did  the  FEAs  for  AMTESS  differ  from  FEAs  for  other  ALL 

training  devices,  if  at  all? 

8.  In  hindsight,  what  would  characterize  an  optimal  FEA?  ALL 


FEATURE  ANALYSIS 


1.  Describe  the  strengths/weaknesses  of  each  of  the  following  ALL 
features  as  concepts,  and  then  describe  how  well  they 
actually  worked  in  the  devices. 

GRUMMAN  Concept  Implementation 

1 .  3D  module 

2.  Student  performance  record 

3.  Instructor  CRT 

4.  Video  disc 

5.  Touch  panel 

6.  Request  help  on  CRT 

7.  Editing  system 

8.  Repeat  lesson  option 

9.  Call  instructor  after 
2  errors 

10.  Performance  feedback 

SEVILLE/BURTEK 

1 .  3D  module 

2.  Student  performance  record 

3.  Student  CRT 

4.  Instructor  CRT 

5.  Slide  projector  unit 

6.  Student  response  panel 

7.  Editing  system 

8.  Remove/replace  capability 

9.  Random  malfunction  selection 

10.  Performance  feedback 

11.  Troubleshoot  only 

12.  Sound  effects 

13.  Instructor  control  panel 
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Target 

ALL 


2.  What  other  features  do  you  consider  to  be  highly 
important? 

-  What  are  their  strengths/weaknesses? 

FEATURE  CONCEPT  IMPLEMENTATION 

3.  For  those  features  you  consider  to  be  ineffective,  what 
would  you  suggest  to  improve  or  replace  those  features? 

4.  How  well  does  the  device  integrate  the  various  features 
to  operate  as  a  system? 

5.  Are  there  training  requirements  which  have  not  been 
addressed  by  the  device  at  all? 

6a.  Why  was  video  disc  chosen? 

6b.  Why  was  a  touch  panel  chosen? 

6c.  What  thinking  guided  the  amount  and  type  of  information 
given  on  instructor  station? 

6d.  What  thinking  guided  the  amount  and  type  of  information 
given  on  the  student  performance  record? 

6e.  Whay  is  programming  written  in  Assembler? 

7a.  Why  wasn't  video  disc  chosen? 

7b.  Why  programmed  in  FORTRAN  IV? 

7c.  Why  is  a  student  response  panel  used  instead  of  a 
standard  keyboard? 

8a.  What  are  the  features  of  the  simulator  that  are  applicable 
to  the  school's  training  course? 

8b.  What  are  the  features  of  the  simulator  that  are  not 
applicable  to  the  school's  training  course? 

8c.  In  your  opinion,  what  features  of  the  simulator 
helped  make  the  lesson  interesting  to  the  students? 

8d.  What  features  of  the  simulator  made  it  more  effective 
than  conventional  training? 

8e.  How  would  you  employ  the  simulator  in  order  to  gain 
maximum  benefit  from  it? 

8f.  What  were  the  features  that  made  this  simulator  easy  to 
operate? 


ALL 

GRUMMAN 

GRUMMAN 

GRUMMAN/ 

S/B 

GRUMMAN/ 

S/B 

GRUMMAN 

SEVILLE 

SEVILLE 

S/B 

SCHOOL 

SCHOOL 

SCHOOL 

SCHOOL 

SCHOOL 

SCHOOL 
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TRAINING  DEVICE  FEATURES  EVALUATION 


We  are  interested  in  your  expert  judgments  about  training  device  simulators 
in  general  and  specifically  about  the  Grumman  and  Sevi 1 le/Burtek  simulators 
used  in  Phase  II  of  AMTESS.  The  judgments  of  a  small  group  of  experts  will 
be  useful  in  preparing  a  report  on  AMTESS  device  features.  This  questionnaire 
survey  has  three  sections: 

Section  1  -  Ranking  simulator  dimensions 
Section  2  -  Ranking  Grumman  device  features 
Section  3  -  Ranking  Seville/Burtek  device  features 


SECTION  1  -  DIMENSIONS 


Please  consider  the  four  dimensions  of  training  simulators  listed  below. 
Next,  rank  the  dimensions  in  relation  to  each  other.  Rank  the  least 
important  dimension  "10."  Rank  the  other  three  dimensions  in  relation  to 
the  least  important  dimension  and  in  relation  to  each  other.  For  example, 
if  you  rank  a  dimension  as  "50,"  it  is  five  times  more  important  than  the 
least  important,  and  only  half  as  important  as  a  dimension  ranked  "100." 
Note:  Your  ranks  have  to  sum  to  100. 


Dimension 

Concept:  is  the  concept  behind  the  feature 
important  in  a  generic  maintenance 
_ training  simulator  or  is  it  unnecessary? 

Implementation:  Does  the  feature  truly  embody 
and  express  the  concept (s) 
behind  it  or  does  it  depart 
_  too  far  from  the  concept(s)? 

Operations:  Is  the  feature  easy  to  use,  does 

it  work  reliably,  or  is  it  unwieldly 
and  frequently  down? 

Motivation:  Is  the  feature  appealing,  does  it 
capture  interest  in  learning,  or 
_ _ is  it  overly  difficult  and  intimidating? 
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For  the  Grumman  simulator,  consider  the  MOS  it  teaches  and  a  typical  trainee 
Please  rate  each  of  the  following  features  of  the  device  along  each  of  the 
four  dimensions.  Use  a  scale  of  0  to  100.  Zero  (0)  is  minimal  and  100  is 
maximal.  You  should  refer  to  the  definitions  of  the  dimensions  presented 
in  Section  1.  Ratings  are  orthogonal  and  need  not  sum  to  100. 

Evaluate  the  Grumman  device  along  the  four  dimensions  below.  Do  not 
compare  it  to  the  Seville/Burtek  device. 


Feature 
1 .  3D  module 


3.  Instructor  CRT 


4.  Video  disc 


5.  Touch  panel 


6.  Editing  system 


7.  Request  help 


8.  Repeat  lesson 
option 


9.  Mandatory 
instructor 
call  after 
two  errors 


10.  Performance 
feedback 


Concept  Implementation  |  Operations  Motivate 


SECTION  2  -  THE  GRUMMAN  DEVICE  -  USAOC&S 


For  the  Grumman  simulator,  consider  the  MOS  it  teaches  and  a  typical  trainee. 
Please  rate  each  of  the  following  features  of  the  device  along  each  of  the 
four  dimensions.  Use  a  scale  of  0  to  100.  Zero  (0)  is  minimal  and  100  is 
maximal.  You  should  refer  to  the  definitions  of  the  dimensions  presented 
in  Section  1.  Ratings  are  orthogonal  and  need  not  sum  to  100. 

Evaluate  the  Grumman  device  along  the  four  dimensions  below.  Do  not 
compare  it  to  the  Seville/Burtek  device. 


Feature 
1 .  3D  module 


3.  Instructor  CRT 


4.  Video  disc 


5.  Touch  panel 


6.  Editing  system 


7.  Request  help 


8.  Repeat  lesson 
option 


9.  Mandatory 
instructor 
call  after 
two  errors 


10.  Performance 
feedback 


SECTION  3  -  THE  SEVILLE/BURTEK  DEVICE  -  USAOC&S 


For  the  Seville/Burtek  device,  consider  the  MOS  it  teaches  and  a  typical 
trainee.  Please  rate  each  of  the  following  features  of  the  device  along 
each  of  the  four  dimensions.  Use  a  scale  of  0  to  100.  Zero  (0)  is  minimal 
and  100  is  maximal.  Ratings  are  orthogonal  and  need  not  sum  to  100. 

Evaluate  the  Seville/Burtek  device  along  the  four  dimensions  below. 

Do  not  compare  it  to  the  Grumman  device. 


Feature 


1.  3D  module 


2.  Student  perform 
ance  record 


3.  Instructor  CRT 


Concept  Implementation  |  Operations  'Motivate 


4.  Slide  projector 


5.  Student  response 


6.  Editing  system 


7.  Student  CRT 


8.  Instructor  con 
trol  panel 


9.  Remove/replace 
capability 


10.  Random  malfunc¬ 
tion  selection 


11.  Performance 
feedback 


12.  Troubleshoot 
only  mode 


13.  Sound  effects 


WVS.V* 


SECTION  3  -  THE  SEVILLE/BURTEK  DEVICE  -  I'SAADS 


For  the  Seville/Burtek  device,  consider  the  MOS  it  teaches  and  a  typical 
trainee.  Please  rate  each  of  the  following  features  of  the  device  along 
each  of  the  four  dimensions.  Use  a  scale  of  0  to  100.  Zero  (0)  is  minimal 
and  100  is  maximal.  Ratings  are  orthogonal  and  need  not  sum  to  100. 

Evaluate  the  Seville/Burtek  device  along  the  four  dimensions  below. 

Do  not  compare  it  to  the  Grumman  device. 


Feature 
1 .  3D  module 


2.  Student  perform 
ance  record 


3.  Instructor  CRT 


4.  Slide  projector 


5.  Student  response 


6.  Editing  system 


7.  Student  CRT 


8.  Instructor  con 
trol  panel 


9.  Remove/replace 
capabi 1 i ty 


10.  Random  malfunc¬ 
tion  selection 


11.  Performance 
feedback 


12.  Troubleshoot 
only  mode 


13.  Sound  effects 


Concept  I  Implementation  !  Operations  1  Motivation 


DIMENSIONS  OF  VALUE  RATINGS 


DIMENSION 

J1 

JUDGES 

J2  J3 

J4 

CONCEPT 

20 

50 

40 

20 

IMPLEMENTATION 

35 

20 

30 

10 

OPERATIONS 

35 

20 

20 

40 

MOTIVATION 

10 

10 

10 

30 

»*  /yyy 


RAW  DATA  FOR  " CONCEPT "  DIMENSION 
SEVILLE/BURTEK  DEVICE  -  USAADS 


1 


w", 

r 

r 


v 


V 


V 

s 

% 

*V 


i 


FEATURE 

Ji 

JUDGES 

J2  J3 

1. 

3D  module 

70 

90 

100 

2. 

Student  performance  record 

80 

100 

100 

3. 

Slide  projector  unit 

60 

50 

50 

4. 

Student  response  panel 

60 

20 

80 

5. 

Student  CRT 

50 

50 

70 

6. 

Remove/replace  capability 

65 

70 

60 

7. 

Random  malfunction  selection 

80 

30 

60 

8. 

Performance  feedback 

60 

90 

100 

9. 

Troubleshoot  only  mode 

80 

70 

60 

% 

% 

\ 

\ 


* 
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JUDGES 


FEATURE 

J1 

J2 

J3 

1. 

3D  module 

80 

95 

100 

2. 

Student  performance  record 

70 

90 

100 

3. 

Slide  projector  unit 

60 

80 

90 

4. 

Student  response  panel 

70 

70 

90 

5. 

Student  CRT 

60 

80 

80 

6. 

Remove/replace  capability 

70 

80 

90 

7. 

Random  malfunction  selection 

75 

70 

90 

8. 

Performance  feedback 

60 

70 

50 

9. 

Troubleshoot  only  mode 

70 

70 

90 

RAW  DATA  FOR  “OPERATIONS"  DIMENSION 
SEVILLE/BURTEK  DEVICE  -  USAADS 


FEATURE 

>i 

JUDGES 

J2  J3 

I. 

3D  module 

75 

70 

90 

2. 

Student  performance  record 

70 

80 

100 

3. 

Slide  projector  unit 

55 

50 

10 

4. 

Student  response  panel 

70 

30 

90 

5. 

Student  CRT 

60 

80 

90 

6. 

Remove/replace  capability 

60 

40 

90 

7. 

Random  malfunction  selection 

70 

50 

90 

8. 

Performance  feedback 

60 

50 

90 

9. 

Troubleshoot  only  mode 

70 

50 

90 

RAW  DATA  FOR  "MOT I  VAT ION"  DIMENSION 
SEVILLE/BURTEK  DEVICE  -  USAADS 


RAW  DATA  FOR  "CONCEPT"  DIMENSION 
SEVILLE/BURTEK  DEVICE  -  USAOC&S 


JUDGES 


.  FEATURE 

J2 

J3 

I. 

3D  module 

70 

90 

90 

2. 

Student  performance  record 

80 

100 

90 

3. 

Slide  projector  unit 

65 

50 

80 

4. 

Student  response  panel 

60 

20 

60 

5. 

Student  CRT 

50 

50 

85 

6. 

Remove/replace  capability 

70 

70 

90 

7. 

Random  malfunction  selection 

80 

30 

75 

8. 

Performance  feedback 

60 

90  ' 

90 

9. 

Troubleshoot  only  mode 

80 

70 

90 

RAW  DATA  FOR  "IMPLEMENTATION"  DIMENSION 
SEVILLE/BURTEK  DEVICE  -  USAOC&S 


JUDGES 


FEATURE 

J2 

J. 

1. 

3D  module 

80 

95 

80 

2. 

Student  performance  record 

70 

90 

90 

3. 

Slide  projector  unit 

60 

80 

70 

4. 

Student  response  panel 

70 

70 

30 

5. 

Student  CRT 

60 

80 

70 

6. 

Remove/ rep lace  capability 

70 

80 

70 

7. 

Random  malfunction  selection 

75 

70 

70 

8. 

Performance  feedback 

60 

70 

85 

9. 

Troubleshoot  only  mode 

70 

70 

85 

RAW  DATA  FOR  "OPERATIONS"  DIMENSION 
SEVILLE/BURTEK  DEVICE  -  USAOC&S 


JUDGES 


FEATURE 

J2 

J3 

I. 

3D  module 

75 

70 

85 

2. 

Student  performance  record 

70 

80 

85 

3. 

Slide  projector  unit 

60 

50 

50 

4. 

Student  response  panel 

70 

80 

35 

5. 

Student  CRT 

60 

80 

85 

6. 

Remo  ve/ rep 1 a  ce  ca  pab i 1 i ty 

75 

40 

75 

7. 

Random  malfunction  selection 

70 

50 

70 

8. 

Performance  feedback 

65 

50 

85 

9. 

Troubleshoot  only  mode 

70 

50 

90 

RAW  DATA  FOR  "MOTIVATION'*  DIMENSION 
SEVILLE/BURTEK  DEVICE  -  USAOC&S 


JUDGES 


FEATURE 

J3 

1. 

3D  module 

70 

70 

90 

2. 

Student  performance  record 

70 

60 

75 

3. 

Slide  projector  unit 

55 

40 

65 

4. 

Student  response  panel 

50 

10 

30 

5. 

Student  CRT 

50 

50 

65 

6. 

Remove/ repl ace  capability 

80 

70 

85 

7. 

Random  malfunction  selection 

75 

20 

85 

8. 

Performance  feedback 

65 

80 

85 

9. 

Troubleshoot  only  mode 

70 

80 

90 

RAW  DATA  FOR  "CONCEPT"  DIMENSION 
GRUMMAN  DEVICE  -  USAADS 


FEATURE 


JUDGES 


1. 

3D  module 

70 

2. 

Student  performance  record 

10 

100 

3. 

Video  disc 

80 

50 

4. 

Touch  panel 

60 

5. 

Request  help 

30 

50 

6. 

Repeat  lesson  option 

20 

50 

7. 

Mandatory  instructor  call 
after  two  errors 

40 

10 

8. 

Performance  feedback 

30 

90 

RAW  DATA  FOR  ’‘IMPLEMENTATION"  DIMENSION 
GRUMMAN  DEVICE  -  USAADS 


1. 

30  module 

50 

70 

100 

2. 

Student  performance  record 

10 

5 

30 

3. 

Video  disc 

40 

20 

100 

4. 

Touch  panel 

30 

50 

90 

5. 

Request  help 

30 

50 

100 

6. 

Repeat  lesson  option 

20 

50 

90 

7. 

Mandatory  instructor  call 
after  two  errors 

30 

5 

50 

8. 

Performance  feedback 

20 

40 

100 

RAW  DATA  FOR  "OPERATIONS"  DIMENSION 
GRUMMAN  DEVICE  -  USAADS 


JUDGES 


FEATURE 


J1 

J2 

J3 

I. 

3D  module 

50 

20 

90 

2. 

Student  performance  record 

20 

20 

80 

3. 

Video  disc 

40 

5 

100 

4. 

Touch  panel 

50 

20 

60 

5. 

Request  help 

40 

70 

100 

6. 

Repeat  lesson  option 

30 

70 

70 

7. 

Mandatory  instructor  call 
after  two  errors 

30 

10 

• 

30 

8. 

Performance  feedback 

30 

80 

90 

t  v  .  .r.  r 


u 


RAW  DATA  FOR  "MOTIVATION"  DIMENSION 
GRUMMAN  DEVICE  -  USAADS 


FEATURE 


JUDGES 


1. 

3D  module 

50 

70 

80 

2. 

Student  performance  record 

10 

20 

10 

3. 

Video  disc 

45 

70 

90 

4. 

Touch  panel 

60 

50 

90 

5. 

Request  help 

30 

50 

90 

6. 

Repeat  lesson  option 

30 

50 

0 

7. 

Mandatory  instructor  call 
after  two  errors 

30 

10 

10 

8. 

Performance  feedback 

30 

80 

80 

FEATURE 


JUDGES 


J2 

J3 

1. 

3D  module 

40 

90 

90 

2. 

Student  performance  record 

10 

100 

80 

3. 

Video  disc 

•  60 

50 

60 

4. 

Touch  panel 

60 

20 

80 

5. 

Request  help 

30 

50 

90 

6. 

Repeat  lesson  option 

20 

50 

90 

7. 

Mandatory  instructor  call 
after  two  errors 

40 

10 

90 

8. 

Performance  feedback 

30 

90 

90 

n-15 


RAW  DATA  FOR  "IMPLEMENTATION"  DIMENSION 
GRUMMAN  DEVICE  -  USAOC&S 


FEATURE 


JUDGES 


1. 

3D  module 

50 

70 

2. 

Student  performance  record 

20 

5 

3. 

Video  disc 

20 

20 

4. 

Touch  panel 

30 

50 

5. 

Request  help 

30 

50 

6. 

Repeat  lesson  option 

20 

50 

7. 

Mandatory  instructor  call 
after  two  errors 

30 

5 

8. 

Performance  feedback 

20 

40 

RAW  DATA  FOR  "OPERATIONS"  DIMENSION 
GRUMMAN  DEVICE  -  USAOC&S 


FEATURE 


JUDGES 


\ 

J2 

J3 

1. 

3D  module 

60 

30 

10 

2. 

Student  performance  record 

10 

20 

90 

3. 

Video  disc 

40 

5 

20 

4. 

Touch  panel 

50 

20 

70 

5. 

Request  help 

40 

70 

90 

6. 

Repeat  lesson  option 

30 

70 

60 

7. 

Mandatory  instructor  call 
after  two  errors 

30 

10 

50 

8. 

Performance  feedback 

30 

80 

50 
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10 

20 
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40 

70 
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4. 

Touch  panel 

60 

50 
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5. 

Request  help 

30 

50 

80 

6. 

Repeat  lesson  option 

30 

50 

40 

7. 

Mandatory  instructor  call 
after  two  errors 

30 

10 

40 

8. 

Performance  feedback 

30 

80 

70 

